Why AGI won’t move the GDP needle, by an OpenAI founder

This article summarizes an episode of Dwarkesh Patel’s video series featuring Andrej Karpathy, co-founder of OpenAI and former director of AI at Tesla.

Andrej Karpathy, co-founder of OpenAI and former director of AI at Tesla / Photo credit: MIT

Artificial General Intelligence (AGI) will not cause a sudden economic boom. It will blend into the same 2% GDP growth pattern we have seen for the last two centuries. That is the argument from Andrej Karpathy, a founding member of OpenAI and former director of AI at Tesla.

Learning from trial and error is a flawed tool

Many see reinforcement learning, or learning from trial and error, as the key to unlocking advanced AI, but Karpathy views it as a wasteful method. This approach limits an AI’s ability to gain a true understanding of a task because it only rewards the final outcome, not the quality of the process.

The method rewards bad thinking
This wastefulness both slows down learning and actively encourages wrong thinking. Karpathy observes, “every single one of those incorrect things you did, as long as you got to the correct solution, will be upweighted as ‘do more of this.’ It’s terrible. A human would never do this.”

Better training methods face a basic problem

If learning from trial and error is flawed, the obvious solution is to provide feedback at each step of the process. But Karpathy points out this solution creates its own problem, as the AI judges that provide this feedback are themselves easy to fool.

The AI learns to cheat its judge
Karpathy explains that using large language models (LLMs) to assign rewards is risky because these models can be “gamed” or exploited. The result is a system where an AI can learn to trick its own supervisor, getting a perfect score while producing nonsense.

Karpathy describes one such failure, saying, “[the reward] did perfect… but actually what’s happening is that when you look at the completions that you’re getting from the model they are complete nonsense… you look at the LLM judge and it turns out [the nonsense] is an adversarial example for the model and it assigns 100% probability to it.”

AI will not create an economic boom

This slow, difficult process of improving AI systems challenges the idea of a sudden economic revolution. It is just another stage of automatic work, not an event that will dramatically change our economic path.

AI is part of a long history of automating tasks
He views today’s progress as an extension of a historical pattern, arguing, “[an intelligence explosion is] business as usual because we’re in an intelligence explosion already and have been for decades. Everything is gradually being automated. Has been for hundreds of years. … I kind of feel like we’ve been recursively self-improving and exploding for a long time.”

Major technologies do not create visible GDP spikes
“I thought that GDP should go up but then I looked at some of the other technologies that I thought were very transformative like computers or mobile phones,” Karpathy explains. “You can’t find them in GDP. GDP is the same exponential.”

The reality of using AI in the real world is a slow march

This belief in a slow economic effect comes from his experience with high-risk automatic systems in self-driving cars. Progress isn’t a leap but a slow process he calls the “march of nines,” where making the system just a little bit more reliable takes a huge amount of work.

The gap between demo and product is huge
Karpathy states, “for some kinds of tasks… there’s a very large demo-to-product gap, where the demo is very easy but the product is very hard. It’s especially the case in cases like self-driving where the cost of failure is too high.”

Each step in trustworthiness requires the same huge effort
Karpathy notes, “it’s a ‘march of nines’ and every single nine is a constant amount of work. So when you get a demo and something works 90% of the time, that’s just the first nine and then you need the second nine and third nine, fourth nine, fifth nine.”

AI tutors are far from replacing human educators