Why RAG is a trap for enterprise AI (and how to escape)

This article summarizes an episode of Decoder with Nilay Patel’s video series featuring Sherif Mansour, head of AI at Atlassian.

Sherif Mansour, head of AI at Atlassian / Photo credit: ADAPT

Generative AI can create text and code quickly, but much of it is what Sherif Mansour, head of AI at Atlassian, calls “slop.” This means the output is correct but lazy. To get better results, he created a plan for making useful AI output inside large companies.

Escaping generic AI output

The best way to avoid generic AI results is to use a method that combines a team’s point of view with its private information and ways of working.

• Taste: A team must add its own opinion and style to its prompts.
• Knowledge: The AI needs to be connected to the company’s own information. This includes private documents, project data, and internal wikis.
• Workflow: The AI must be used inside a specific work process. This makes the AI a part of how work gets done, not just a separate tool.

The universal but worst interface

To use this plan, companies must think differently about how people use software.

Rethinking the chat window
The growth of large language models (LLMs) has made chat windows the common way to start using AI. It is easy to use, but people are now asking if it is the best long-term option.

Mansour argues, “I always go with the phrase of, like, ‘Well, what’s the universal interface to any LLM?’ It’s conversation… I do believe and I would argue, I think we’re already seeing it, that chat is not the universal interface to all AI.”

A lesson from the command line
To understand the future of how people use AI, Mansour suggests looking at the past. The tech industry has seen this same problem before with general tools versus specific ones.

Mansour explains, “If I could rewind back to the MS-DOS terminal days… the terminal was the universal interface to the operating system… But we learned very quickly that it was actually the worst interface for some use cases. And so, over the years, we built verticalized apps on top of the terminal.”

When one size fits none
Making people use one text-based tool for every task was difficult and limiting. This led to new kinds of tools.

He continues, “We learned very quickly that [the MS-DOS terminal] was actually the worst interface for some use cases. And so over the years we built verticalized apps on top of the terminal to do word processing, image generation, spreadsheets, audio, podcast recording, etc.”

Why getting data fails in large companies

Getting data is another challenge. Common ways of doing this, like retrieval-augmented generation (RAG), face special problems inside large companies. RAG works by retrieving relevant documents to help the AI answer questions, yet this approach often struggles with the complexity of internal business data.

The permissions problem
Mansour states, “RAG gets incredibly hard in an enterprise context because everything is permissioned, even at the field level… I might be able to see some fields, and you might not be able to see other fields. And so, there’s a very large investment of teams just making sure it has a semantic index behind it.”

Mapping how work is connected