Inside the work of AI systems engineers

Training large AI models grabs headlines. Running them in production is a different engineering challenge altogether.

Once a model is trained, it still needs to respond quickly, reliably, and at a cost that makes large-scale use viable. That work falls to machine learning (ML) systems engineers, who optimize everything from GPU performance to how requests move through inference infrastructure.

Rachel Lim (right), an engineer who works on AI systems / Photo credit: OpenAI

Rachel Lim, a Singaporean engineer who works on AI systems at OpenAI, works on that layer of the stack for tools like ChatGPT. She spoke with Tech in Asia about what AI systems engineers actually do and how aspiring engineers can prepare for roles like hers. This interview has been edited for clarity and brevity.

When people ask what you do, how do you explain it?

I usually say: I make computers faster and more affordable, so more people around the world can use our models seamlessly through ChatGPT and our API.

To dive deeper, I’m on the inference team, which is responsible for several things. We work on making our AI systems fast, reliable, and able to handle millions of people using them at once. That means measuring how the system performs, finding where the bottlenecks exist in our system, and improving them so ChatGPT runs smoothly.

See also: How to break into robotics without an engineering degree

As engineers, we think about this on the meta level too: Where are our own processes slow? What can we improve to make our systems more reliable? What parts of our workflow can be automated? We also keep an eye out for new model capabilities coming down the line and think about how to enable them.

How is ML systems work different from model research?

We focus on different parts of the problem. In ML systems, we tend to be closer to the hardware and have to understand deeply what computation is actually running – every single matrix multiplication or operation, how data moves through the system.

In research, people think more about the why behind every computation – the architectural innovations, training algorithms, and statistical principles that underlie the choices that lead to improvements in model capabilities and intelligence. Research and systems teams have to work together to build models that are useful, and many of the best researchers are also strong engineers.

For someone who wants to break into AI systems engineering, what skills are the most important?

Clear thinking and strong engineering fundamentals will matter most. Systems thinking is especially important as well, to understand how different parts interact at scale. It also helps not to be intimidated by math like linear algebra, since it shows up quite often.