Inside Flipkart’s monster-cruncher: how it gleans insights from a petabyte of data daily

Flipkart CEO Kalyan Krishnamurthy with his team. Photo credit: Flipkart Stories

Gender is complicated for the data scientists at Flipkart, India’s leading ecommerce site. It’s not enough, for example, to know that the shopper is female. What if she’s shopping for her husband today? So, apart from the label she chose while signing up for the account, there is also a behavioral gender.

This is gender based on your behavior, whether you tend to shop like a female or a male when using your account. The behavior can also vary from session to session, so there’s a third in-session gender.

“We have to compute the label gender when we send you survey results so that we can address you properly and all. But when we have to show you what you want, we will piggyback on the behavioral gender and [adjust that] as soon as we know a little bit about this session [and] what your mood is today,” says Sandeep Kohli, Flipkart’s senior director of engineering.

All that work is just for one user. To see the scale at which Flipkart is doing this, consider the following numbers:

Data credit: Flipkart; chart by Tech in Asia

And gender is just one parameter for customer insights. All kinds of behavioral, demographic, and usage data go into making product recommendations or trying to ensure a customer doesn’t drop out without completing an intended purchase.

During the recent Big Billion Day sales, for example, pageviews in a day rose five-fold to nearly a billion, says Kohli. “You have to accommodate that [spike] without degrading the experience because people don’t have too much patience on such days. So every second of delay costs us a few hundred customers who could drop off at that point.”

Survival of the smartest

To manage that kind of data and do advanced analytics and personalization, Flipkart has built its own data center – the only Indian internet company to have done so. And that takes a lot of hardware. “We have 5 petabytes in RAM, 120 petabytes of disk storage, and [a] tremendous amount of cross-sectional bandwidth, because anything can become a bottleneck at that kind of scale,” says Kohli. “It is extremely important to know about your customers and serve them better, because at this point in the highly competitive ecommerce industry, it’s a survival game.”

If you record HD video 24/7 for 3.4 years, you would reach a petabyte.

Every action on the site involves analytics. Take for example a simple search query: how far did the user have to scroll down? If the search results are relevant, the user should be able to find the product they’re looking for without needing to scroll too far down. Now imagine getting the search right for 100 million users.

Efficiency also plays a vital part in customer experience. Where a product order comes from can help with inventory planning, delivery time, and so on. Then there are insights that are important for running the business, such as figuring out if product ratings are genuine or whether somebody is gaming the system for sales offers.

Flipkart gets 10 terabytes of user data each day from browsing, searching, buying or not buying, as well as behavior and location. This jumps to 50 terabytes on Big Billion Day sales days. There’s also order data, shipping data, and other forms of data captured by different systems. All this is mixed together and correlated for meaningful insights. “We actually process more than a petabyte of data every day in order to make sense of what is happening at our scale,” says Kohli.

Following the breadcrumbs

Doing it at scale