They set out to sell an AI bot and ended up plugging massive data loss

Image credit: Pixabay.

They chanced upon a humongous problem while chasing another.

Dipankar Sarkar and Akshaya Aron set out to build a bot with enough artificial intelligence to automate marketing. That’s a tough problem – not just to solve but also to explain to the rest of the world. There are, after all, several marketing automation tools out there.

The term ‘marketing automation’ is used loosely for almost any piece of technology that can help marketers do their jobs better. Usually, the high-brow ones involve a few humans tweaking codes to segment users into neat profiles and automating the system to identify and target a certain profile with a certain marketing message. Dipankar and Akshaya wanted to take this a few steps ahead.

They built an AI bot – Octo-matic – that does not require any human intervention after you integrate it with your app. It will help you reach each of your users personally, analyze their behavior, and act on them automatically. But the bot needs to be fed raw data of your user behavior to do its job. The more raw data you have in store, the smarter the bot will be.

But when, Dipankar and Akshaya tried to deploy Octo with a bunch of companies, they stumbled on something huge.

“We wanted to show the companies what our bot could do. So we went up to them and said, ‘we need three months’ data or as much data as you have from your end so that we can actually do a good job of delivering you full automation.’ But none of them had any!” Dipankar says.

Though all these companies were gathering a lot of user behavior data, they were not storing it. They had installed data analytics tools like Google Analytics (GA), Mixpanel, or CleverTap to make sense of the data. “So all their raw data would go to these tools. The marketing teams were left with just processed information on their dashboards,” he says.

A lot of things need to be built before better bots come, before better user personalization happens, and so on – we want to fuel that.

Google Analytics, Mixpanel, and such tools process data, chop them up into neat counters, and serve them up to you. So, your historic data have been sliced, diced, and baked so that now you have tables and numbers to act upon. “But you can’t do any future analysis on the raw data as you’ve lost it,” Dipankar points out. “GA and others have no incentive to store your data.”

But you have. It’s your data and it’s valuable for your business. “Tomorrow, if your company decides to do user personalization at a high level, with very interesting algorithms that look at historical data and generate specific content for each user, that’s when you realize that you don’t have the data set,” Dipankar explains.

That was an interesting problem – and a big problem, he and Akshaya figured. So they decided to tackle it.

“Somebody has to collect all that raw data. Because that is the first step to building something intelligent out of data – you need raw data.” So they built a middleware to allow web and mobile companies to copy and store raw data. And they open-sourced it.

Why open source?

The biggest requirement for any form of machine learning or AI system is high quality historic data. “From our point of view, to actually build intelligent services it would require a whole new layer in the existing software development stack where all analytical data would be stored – as early as possible in the application’s lifecycle. We decided that our first goal should be to get people to store this data! This is how we decided to open-source that part of our stack,” Dipankar explains.

The AI obsession

Clearing the path, digging the road