The race to crack AI’s cost problem

Welcome to The Prompt, your Monday dive into the world of AI that puts Asia front and center. From Big Tech’s power players to the region’s scrappy disruptors, we cover it all. Want full access to all our AI reporting? Subscribe to us here.

---

Hello reader,

Every major AI lab is essentially renting the pipes that run their business. That’s starting to bother more and more of these companies.

One example is OpenAI, which recently teamed up with Broadcom to build Jalapeño. It’s the lab’s first custom chip, built specifically for large language model inference, the process of actually running its models for millions of users every day.

It seems the Silicon Valley giant may be getting tired of paying premiums to run its software, so it is trying to own the physical pipeline as well.

And OpenAI is not the only company that feels this way.

Google is already deploying its eighth-generation TPU 8i and 8t custom silicon, while Amazon is scaling its custom Trainium3 hardware with US$225 billion committed into it. There’s also Meta, which has dropped successive generations of its MTIA chips.

What does this mean for the individuals and businesses that are clients of OpenAI? Well, computing costs won’t instantly drop just because Jalapeño exists.

Presumably, costs will only go down for end users if OpenAI decides to pass its potential savings from the chip to the ecosystem by slicing its public API token pricing.

Likewise, this in-house chip rush won’t cool down the market for general-purpose graphics processing units (GPUs), which are crucial for model training, anytime soon.

This isn’t to say that focusing solely on inference won’t make much of a difference when it comes to costs. The process comprises two-thirds of total compute spend, according to aggregated cloud platform Spheron.

However, given the recent race to lower token prices, even as compute continues to get more expensive, OpenAI passing down these savings is likely to happen.

The AI firm’s effort to enter the hardware race is telling – the costs of AI hit everyone hard, from frontier labs to regional startups, which is the thread running through this edition.

---

TOP AI READS FROM OUR DESK

---

WHAT WE THINK

---

FOUNDER FOCUS

---