AWS engineer warns caching traps your system in a broken state

This article summarizes an episode of Ryan Peterman’s video series featuring Marc Brooker, VP and distinguished engineer of agentic AI at AWS.

Marc Brooker, AWS VP and distinguished engineer of agentic AI / Photo credit: Marc Brooker

Marc Brooker, a distinguished engineer at AWS, views long-term on-call rotations as a massive career advantage rather than a chore. Daily operations expose harsh truths that clean planning meetings often miss. This hands-on reality changes how engineering teams must handle postmortems, caching, and AI tools.

To prevent expensive mistakes, engineers must stay intimately connected to how their systems actually run in the real world.

The advantage of the pager path

Large systems fail in chaotic ways that perfect planning cannot predict. Companies need a mechanism to bring this reality into their leadership decisions, and staying on-call provides exactly that feedback loop.

Brooker explains, “The majority of my in-practice knowledge about how to build distributed systems has come from being on call and analyzing and deeply understanding these postmortems. Without grounded knowledge of what it actually means to run and understand systems, you miss how systems really behave.”

Using confusion to build better tracking
When teams review an outage, they must strip away their biases to find the real weaknesses.

“A really great postmortem [starts by] getting into the details,” Brooker argues, “and making sure that you deeply understand what happened rather than assuming what happened. If you cannot understand what happened, that teaches you something about your logging and metrics.”

The cost of constant firefighting
Companies often celebrate the “hero” engineers who step in to fix outages, but this creates a misleading sense of stability. Normalizing these operational heroics feels like strong ownership, but it means the team is stuck in a break-fix cycle.

Brooker insists that teams must stop spending expertise on short-term fixes and invest in permanent improvements to system reliability.

The hidden risks of caching

Time spent running live systems changes how teams build software. While engineers often view temporary caching as a harmless way to boost speed, a look at how systems recover tells a much deeper story.

“There is a mode where the cache is full of the right data and the system is fast and healthy,” Brooker notes. “There is a mode where the cache is empty or contains the wrong data and the system is slow, often down, because the backend is not scaled to deal with all of this uncached traffic.”

Trapped in a broken state
Because once an app crashes, engineers expect it to recover naturally when the traffic spike ends. However, certain design choices trap the system in a broken loop. Brooker explains that in a bad state, the system might remain down because the traffic overwhelms the database or network, preventing the system from ever refilling the cache and recovering on its own.

While individual developers may rarely encounter full system crashes, an industry-wide look reveals otherwise.

Designing for guaranteed recovery

The changing definition of engineering value