What is big data anyway?

Many people talk about big data these days. Recent startups are moving towards exploring big data. Current issues in research also deal with deep learning, which is basically a way to exploit big data. It is clear we are now facing the information revolution. There is a very huge growth of data, as in the last 2 years there are around 2 ZB (Zetabyte = 1 billion Terrabyte) data generated in this world. This enormous flow of data is mainly because of the extensive use of server logs, IoT (Internet of Things), sensors, social media, and emails.

How big is the big data?

If you think your laptop with 10 GB hard disk already stored big data, it is still way too small. A single server of database 100 GB is not even close to what so-called “big data”, even when we make it 11 TB shared of database in distributed system. But, a 100 TB of MPP (massively parallel processing) is quite close to big data. Then, you can truly have a big data platform at 1000 TB (=1 PB, Petabyte). But it is not enough, we expect to have a cutting edge of future big data platform at 1 EB (Exabyte = 1 million TB).

Big data brings together a new technology and a new philosophy to make a great business transformation.

So what are the new technologies that big data offer?

1. Volume

Instead of doing the scale-up architecture to squeeze more hardware into the servers that can cost more money and lower performance, big data relies on scale-out architecture to increase capacity. It means, we add more servers to the network so that it is distributed. This infrastructure with distributed file system introduces a huge degree of parallelism with lower predictable cost compared to scale-up architecture.

2. Variety

Fact shows that 80% of enterprise data is unstructured. Yes, most of the time, we also process abstract and unstructured data in our daily life. Conservative way of processing the data with high volume and variability is to reduce or reshape it so that it can be handled safely by the computer (for the sake of computation time). You can leave that conservative way and just process the original data without losing some of the truncated data after being reshaped. Yes, of course big data can handle this original variety.

3. Velocity

Since big data does not need sampling, it can be quicker. We need that speed to catch data. No sampling needed also means less complex model to use. This answers the problems of accuracy and accountability in handling data. When we use more data with limited resource (let’s say.. a-not-so-hi-spec laptop), we re-sample the data which may lead to a lower accuracy. Vice versa, if we need to increase the accuracy but again our computer is not capable to handle more data then we propose a more complex model to deal with it, which unfortunately will end up with lower accountability.

More data trumps smarter algorithms. (from Google)