Let’s analyze Ashley Madison sexual data to understand big data

Why you’re reading this article

Harvard Business Review calls it a “management revolution”. McKinsey released a whopping 156 page report touting it as “the next frontier for innovation, competition, and productivity.” Palantir, a startup that used it to help the US government track down Osama Bin Laden, is now one of the hottest companies in Silicon Valley valued at $20B based on their latest funding round. Forget Google, Facebook, and Twitter, bright college grads have already shifted their sights set on Palantir. Big data has become the new black.

The big data wave isn’t just creating companies slated for multi-billion dollar IPOs and exits, it has also created new job opportunities. What used to be a boring number crunching chore is now called data science, which Harvard Business Review coined the “sexiest job of the 21st century”. Every cool startup now boasts a data science team led by some chief data scientist. As usual, digital agencies are jumping on the bandwagon with some of them creating new units that supposedly “bring together data sciences, social, new age content, and emerging marketing technology with sound business thinking to create a proposition that’s truly integrated.” Whatever that means.

“There’s gold in the streets, just waiting for someone to scoop it up.” – Walter White in Breaking Bad

Using big data to look at big data, Google shows that search volume for ‘big data’ follows a nice hockey-stick trajectory envied by many startups. It’s pretty clear – big data is big business.

The REAL Reason why you’re reading this article

“Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…” – Dan Ariely

Despite all the media hype about big data, the sad reality is that no one actually truly understands it. What’s causing all this misunderstanding and why are we long overdue for a paradigm shift in big data?

Mythbusters: Debunking Big Data myths

1. Big Data requires an expensive enterprise platform

The biggest lie in big data is that it’s complicated and tech-heavy. It’s also the primary reason why companies fail to adopt big data. We’re talking specifically about fear of technology and/or obsession with technology. The notion that big data requires a tech-savvy professional puts off a lot of people and prevents them from taking initial baby steps towards working with data in their organizations. On the other hand, there are those that focus too much on technology for technology’s sake. Big data platforms are a means to achieve business goals, not an end in itself.

Peter Thiel, the billionaire venture capitalist who founded Paypal and Palantir, argues against over-emphasizing technology. In his bestseller Zero to One: Notes on Startups, or How to Build the Future, Thiel says that “we’ve let ourselves become enchanted by big data only because we exoticize technology. We’re impressed with small feats accomplished by computers alone but we ignore big achievements from complementarity because the human contribution makes them less uncanny.”

Only until people see through this smokescreen of big data complexity created by what we’d like to call the “Big Data-Industrial Complex” – the sum of companies with three-letter acronym names that peddle big data technology products – we’ll be able to move on towards addressing the real challenges of big data.

Doing a simple search on Google for ‘big data’ shows how competitive this space is and how much money is at stake for the Big Data Industrial Complex.

Ashley Madison leak reveals if bigger is better…

A few parting words