Why token prices are the wrong way to judge AI

This article summarizes an episode of 20VC with Harry Stebbings’s video series featuring Eno Reyes, co-founder and CTO of Factory.

Image credit: Timmy Loen

Cheap AI can cost companies more in the long run when employees must spend time fixing mistakes. Eno Reyes, co-founder and CTO of Factory, says that judging systems by token prices is misleading.

To succeed, businesses should focus on the final cost of a task, move most workflows to open models, and build their own software to protect company knowledge.

Token pricing hides the true cost of finished work

To gauge AI expenses, leaders must abandon upfront token fees and measure the financial impact of a verified result.

Reyes notes that “how much a code review costs is far more interesting than how much the tokens inside that code review cost.”

Organizations can compare these systems by clarifying the desired work before testing vendors:

• Business definition: Set clear grading criteria before selecting a model.
• Accuracy workflows: Create verification methods before company-wide deployment.
• Retry tracking: Monitor how many attempts a system needs to complete a single job.

Open models dominate the volume market

Once businesses measure task costs, they will shift chores to open models to stay within budget:

• General tasks are moving to open systems to reduce reliance on single vendors.
• Using top-tier AI for chores wastes capital.
• Relying on open options for high-stakes projects creates failures.

Reyes predicts that “in three years, 99% of workflows are going to be done on open models, but 1% of those tasks is probably going to represent 30% to 40% of the economic value of the future of intelligence.”

Software harnesses secure institutional knowledge

As fewer tasks rely on frontier models, the software layer surrounding these systems becomes the control center. Reyes calls this management layer a harness, which empowers internal teams to direct workflow details that AI systems cannot handle alone:

• Direct choices automatically. Route tasks to cheaper options based on results.
• Save project progress. Prevent workflows from restarting from scratch.
• Record successful methods. Store problem-solving tactics as business knowledge.

Keeping this technology internal guarantees that vendors do not own the record of how a business operates. Hosting software locally gives companies the leverage to walk away from bad deals when market conditions change.