Nvidia CEO: Why a cheap AI chip can be an expensive mistake

This article summarizes an episode of Bg2 Pod’s video series featuring Jensen Huang, CEO of Nvidia.

Nvidia CEO Jensen Huang / Photo credit: Shutterstock

Nvidia CEO Jensen Huang believes a chip’s true value comes from its long-term performance, not its initial price. He argues that the old way of relying on all-purpose processors is giving way to systems designed specifically for AI. In this new world, the total cost of ownership matters more than the upfront cost of a single component.

Moving beyond the CPU

The traditional approach to building data centers has hit a wall. The problem is imposed by physical limits, not a temporary slump in the market. This requires a new approach to building and investing in data centers.

A performance dead end
For decades, engineers could count on CPUs getting predictably faster and more efficient. That progress has stalled. As Huang puts it, “General purpose computing is over… the future is accelerated computing and AI computing.”

He sees this transition as an opportunity for partnerships, noting that “general purpose computing needs to be fused with accelerated computing.”

A trillion-dollar refresh
This change means that the world’s existing data centers are effectively out of date. Replacing them is a massive project. “There’s trillions of dollars of computing infrastructures in the world that has to be refreshed,” Huang explains. “And when it gets refreshed it’s going to be accelerated computing… moving into AI, that’s hundreds of billions of dollars.”

Why AI needs more power than ever

The demand for this new infrastructure is growing for several reasons. The initial power consumption for training AI models was only the beginning of a much larger trend.

More than just training
Early predictions about AI’s computing needs focused mostly on the power-hungry process of training a model from scratch. That turned out to be a limited view. “I underestimated,” Huang admits, explaining that there are now three major phases, each demanding its own huge investment in hardware.

• Pre-training: The initial creation of a large language model.
• Post-training: Refining a model by letting it practice a skill, which combines training with real-time use (inference).
• Inference and “thinking”: Moving from simple, one-shot answers to multi-step reasoning, where an AI thinks, researches, and revises before giving a final response.

From quick answers to deep thought
Early AI was like a simple search query: you asked a question, it gave one answer. Now, AI models are expected to “think” through problems, which uses far more processing power for each request.

Huang details the new process: “So think before you [AI] answer… The longer you think, the better the quality answer you get. While you’re thinking, you do research… you learn some things, you think some more… and then you generate an answer.”

Why a free chip can be more expensive

Nvidia’s advantage comes from its total system performance rather than the sticker price of its chips. By focusing on what it costs to get a result, including electricity and infrastructure, Nvidia makes its platform a better investment, even if competing hardware is cheaper.

The key metric is performance per watt. Data centers have a fixed power budget, and the goal is to generate as much value (or “tokens”) as possible from every watt. A cheaper but less efficient chip wastes that limited power, along with the physical space and cooling systems that support it.