The myth of the set and forget AI agent

This article summarizes an episode of SaaStr AI’s video series featuring Jason Lemkin and Amelia Lerutte from SaaStr.

Amelia Lerutte, chief AI officer of SaaStr (left) and Jason Lemkin, CEO and founder of SaaStr (right) / Photo credit: Amelia Lerutte

SaaStr founder and CEO Jason Lemkin says AI agents are creating new work instead of making things easier. The company has been dealing with the daily challenges of managing these agents, pushing back against the common belief that automation can largely run on its own.

The company understands this problem by treating its AI agents like employees. Lemkin believes they need direct management. By checking on them daily, the team can avoid the big problems that affect companies that are not ready.

Rethinking the digital workforce

What automation promises is often not what companies get. While leaders expect software to run on its own, these new AI tools need to be managed differently. Treating them like normal software causes the system to break down immediately.

Amelia Lerutte, SaaStr’s chief AI officer, explains the change in thinking needed. “You have to almost think about [managing AI agents] now… as managing 20 different people,” she explains. Her warning shows that a system you don’t watch falls apart fast, and she warns that waiting a week makes an agent useless.

“Your agents are just going to idle if you don’t check in with them every day,” Lerutte explains, calling it a waste of time and money. These AI tools need constant adjustment to work on changing goals.

Dealing with a broken system

The work of checking agents daily is made harder by technical limitations. With no single place to manage all the different agents, managers have to use many separate apps.

Lemkin points out that companies still lack a unified control panel. “We haven’t found anyone that can integrate AgentForce, Artisan, Qualified, Monaco, Momentum… there’s nothing out there today… that at some magical API or other level can integrate everything for you. That product does not exist.”

Having so many different tools forces people to keep switching between programs, which is tiring. “They don’t all function the same way,” observes Lerutte. “They all speak slightly different languages.” This problem wastes time and forces people to do the same boring tasks over and over.

The danger of a single point of failure

As companies build more complex automations, knowledge often becomes concentrated in one person. In many cases, a single employee ends up managing the prompts, connections, and system logic behind the agents.

Lemkin points out the risk this creates, even at their small company. “If you left for some reason, we would have to cease using our agents. There’s no plan B,” he says, addressing Lerutte directly. “We’re just all learning that you need some sort of chief agent officer.”

This is a common problem in the industry. Talking about another company, Lemkin saw how all the knowledge ended up with the one salesperson who was best at using the new tool. Basing your sales system on one person’s knowledge is very risky.

Planning for someone to leave

When only one employee understands how to manage a company’s AI agents, passing that job to someone else is incredibly difficult. Standard training manuals fail because the system relies on unwritten commands and the manager’s personal style rather than formal procedures.

The real cost of growth