Big data is evolving and it’s got its sights on you

“What if you had the power to know what the world was thinking about at any time?” That was the question posed by Chris Moody, vice president of data strategy for Twitter, at the start of the Structure Data conference in San Francisco.

The answer to his question was clear: bring it on.

Photo credit: Negativespace.co.

Huge data used to predict

Indeed, there are machines that can now collect huge amounts of data from connected users every second of the day. These give companies the ability to observe, analyze, and learn about what much of the world is doing and thinking. Welcome to machine learning and the global microphone.

Millions of users who use Twitter for communication around the world are giving the company a daily window into the global conversation.

According to Chris, Twitter has archived every single tweet in a fully-searchable database since Jack Dorsey sent the first message ten years ago. This gives them an enormous amount of rich, real-time data that, perhaps, is the company’s single most valuable asset.

Twitter has archived every single tweet in a fully-searchable database.

As Structure Data’s conference showed, Twitter is not alone in seeking to exploit the data they have. Facebook has so far logged 2 trillion posts. According to Alan Packer, Facebook’s director of engineering for language technology, they intend to use these carefully mined data to keep their users happy.

This means that when a Facebook user conducts a search, they might receive results analyzed by the company’s machine learning algorithms. This can, for example, distinguish a good restaurant from a bad one using millions of customer comments.

As machines get increasingly smarter from digesting huge amounts of data, they can now understand more than user interest—they can also recognize intent.

According to Alan, Facebook has built a way to understand when people plan to travel. It does this by analyzing posted queries to friends like “I’m going to France and was wondering what I should see in Paris?”

The value of this kind of data-driven analysis is obvious. “Advertisers are pretty interested to know that you are going to Paris,” said Alan.

Even a news service site, used by countless millions around the world, offers the ability to scrutinize reading habits and derive valuable data for platform owners.

Suju Rajan, director of research for personalization science at Yahoo, described how Yahoo relies on sophisticated analytics to determine how long users read a particular story. So if you frequently read stories about Justin Bieber, don’t be surprised to see additional content about the music star suddenly appear on your Yahoo home page.

Big data in different industries