How Google & OpenAI scale AI autonomy without losing trust

This article summarizes an episode of Lenny’s Podcast’s video series featuring Aishwarya Naresh Reganti and Kiriti Badam, AI product engineers from OpenAI and Google.

Image credit: Arsal Ysfin

Aishwarya Naresh Reganti and Kiriti Badam, AI product engineers from OpenAI and Google, believe we are in a new age for building software. Building the software, once the most expensive part of the work, is now cheap. The cost is now in the design phase. This reversal of old software costs forces companies to rethink how they plan products and shows the cost of bad planning.

Autonomy creates a new kind of risk

Building older software is like following a set of directions. With AI systems, that certainty is gone, which breaks old ways of working.

The end of full control
In the past, developers dictated the rules. With AI, that certainty is gone. Reganti points out that teams are swapping predictable code for a system where three specific things are unknown: the user’s behavior, the AI’s reaction, and the logic connecting them.

“The first difference that most people tend to ignore is the non-determinism,” Reganti notes. “You’re now working with an input, output, and a process. And you don’t understand all three very well.”

Autonomy requires trust
The promise of AI is in letting it do difficult tasks on its own. But for product teams, giving up control makes it harder to keep the product safe and reliable for users.

Reganti says, “Every time you hand over decision-making capabilities or autonomy to agentic systems, you’re relinquishing some amount of control on your end. When you do that, you want to make sure that your agent has earned your trust.”

Start small, scale autonomy gradually
Earning that trust can’t happen overnight. It requires a step-by-step process that builds user trust over time, instead of assuming it exists.

Badam advises, “You need to deliberately start in places where there is minimal impact and more human control so that you have a good grip of what the current capabilities are. Then you slowly lean into more agency and less control.”

Leaders must become hands-on learners again

This new risk makes old instincts unreliable. Successful leaders have to relearn how to think from first principles.

• Leaders must use the technology themselves.
• They must accept that their old instincts may be wrong. AI forces leaders to relearn what is possible and what is difficult.
• They must be willing to learn from everyone. This requires enough openness to admit they may be the dumbest person in the room.

Your experience is now a liability
In this new situation, the danger is not knowing what you do not know. Reganti argues, “You must be comfortable with the fact that your intuitions might not be right. You probably are the dumbest person in the room, and you want to learn from everyone.”

Difficulty is the new advantage

This way of thinking creates a strong work environment. Teams that succeed accept the difficulty of building AI systems as an advantage.

A way to build unpredictable systems

The term ‘eval’ has lost its meaning

One approach is never enough