What if your phone could read your face? This ‘mad’ couple brings artificial intelligence to the smartphone camera

After their baby fell asleep, Ashwini Asokan and Anand Chandrasekaran talked. Plonked on a sofa, they discussed artificial intelligence, endlessly, late into the night. The couple are fascinated by the science of machines that see – computer vision, in other words. Could this sophisticated technology be brought to smartphones, even low-end ones, and used to create a compelling experience for the mobile user? This was the dream that kept them awake through the nights in the heart of Silicon Valley.

Asokan was leading the mobile innovation team at Intel’s Interaction and Experience Research Lab (IXR). Her team of designers, anthropologists, and engineers were exploring the future of mobile technology. As for Chandrasekaran, he had just completed his post-doctoral studies in neuromorphic engineering – building computer chips that imitate the human brain – at Stanford University, and was working on neuroscience-based projects as a consultant. During his three years at Stanford, he was in the thick of artificial intelligence. He was part of the team that built Neurogrid, a system that mimics a million neurons in real-time. He also designed and simulated a chip that could mirror the plasticity of the brain – its ability to change with experience.

The couple had enough grey matter to work on their dream. It would take them a year or two to build the technology, they estimated.

At the time they were ideally located in Silicon Valley, which is the hotbed for tech startups. But it’s expensive. There was a baby and a toddler at home as well. So, after much debate, the couple decided to pack their bags, return to India, and start up in their hometown, Chennai, where they had a family support system and their savings would give them a longer lead time.

In less than a year from then, the couple’s startup, Mad Street Den, is up and running with a cloud-based platform that uses artificial intelligence to enable any smartphone with a camera to identify faces, detect facial expressions and emotions, and react to facial and head gestures. The expression acts as a trigger for a certain action – for example, frowning at the phone when an unwanted call comes in could make it shut down forthwith, or lifting an eyebrow could send the caller a message asking ‘What now?’

This image-recognizing platform, called MAD Stack, can be used by app developers and companies to create a futuristic mobile user experience. “The idea is to make machines more useful by making them a bit human: fun, intelligent, and relevant. We use computer vision to do that,” Asokan tells Tech in Asia.

The process of recognizing a human expression or responding to a gesture is simple for a human brain, but quite complex for a smartphone camera to do digitally. It is artificial intelligence that enables a camera to do this. What’s more, the app keeps getting smarter with use, through machine learning algorithms.

When machines can read your mind

Computer vision is a much-maligned technology. Science fiction, on paper and on screen, made sure that it’s as much feared as it’s glorified. From a nebulous Big Brother snooping on citizens to freaky Transformers that are aliens disguised as machines. This irks Asokan. “This technology can have countless fun uses. Why should lay people hear of it in a negative light mostly? Anand and I want to change that,” she says. “Our goal is to move the conversation around computer vision away from surveillance, security, and all the scary stuff to make it fun.”

She cites an array of applications for this technology, some of which are already out there: Amazon’s new Fire Phone can recognize barcodes, box art, or even TV audio, and help you find the object of your desire. If a phone can recognize a barcode, why not a child’s facial expression? And how might that be used in a fun game?

As for businesses, such a capability could take customer analytics to a new level. Amazon’s Firefly, billed as “visual search on steroids”, is only the beginning. Mad Street Den promises to bring similar visual search technology to any smartphone. Its object recognition feature could be a neat trick for Indian ecommerce companies like Flipkart, Snapdeal, and others competing with Amazon.

The Mad Street Den team has built its platform keeping these applications in mind.

From gaming to weeding

An eye on the competition