How Grab built a platform to conduct experiments more effectively

Photo credit: scyther5 / 123RF

This article was co-authored by Abeesh Thomas Stephen (technical writer at Grab) and is part of Tech in Asia’s content partnership with Grab. Visit their engineering blog here.

At Grab, we constantly experiment to improve our app’s user experience. But running many experiments can be messy, complicated, and expensive. This is why we created the Grab Experimentation Platform (ExP), which enables us to run controlled experiments to test a new feature before we roll it out. The goal of ExP is to make sure that new features roll out without any hiccups and that causal relationships are analyzed correctly.

Experimentation platform portal

Why we built ExP

In the early days at Grab, experiments were performed on a small scale. Users defined metrics, and then computed and surfaced those metrics for a small set of experiments.

The process was rather painful. When product managers wanted to run an experiment, they would set up a meeting with product analysts, data scientists, and engineers. Experiments were designed, custom logging pipelines were built, and services were modified to support each new experiment. It was an expensive and time-consuming process.

To overcome these challenges, we wanted to build a platform with the following goals in mind:

• Create a unified platform that prevents multiple concurrent experiments from interfering with one another, and allows engineers and data scientists to work on the same set of tools.
• Allow simple, fast, and cost-effective experiments.
• Automate the selection of representatives to perform A/A testing.
• Support power analysis to perform appropriate significance tests.
• Enable a fully automated data pipeline where experimental data is streamed in real time, then tagged and stored in S3.
• Create a platform for plugging in custom analysis modules.
• Create event triggers/alerts on important business metrics to identify adverse effects of a change.
• Design a centralized online UI for creating and managing experiments. (We are constantly improving this. The long-term vision is to allow anyone in the organization to create and run experiments.)

Since implementing ExP, we have seen the number of experiments grow from just a handful to about 25 running concurrently. More impressively, the number of metrics computed per day has grown exponentially to about 2,500 distinct metrics and roughly 50,000 distinct experiment/metric combinations.

At this scale, we encountered some issues. Here’s the architectural approach we used to address them:

Prevention of network effects

We have several types of users: driver partners, passengers, and merchants. Unlike most experimentation platforms that deal with a single web visitor, our users interact with each other, which sometimes leads to network effects. For example, an experiment on promotions can lead to a surge of demand relative to supply.

Control and treatment assignment strategies

Prevention of experiment interference

Components of ExP

Structured experimental design

Sample experiment