Big data and Asia: the trends and the challenges



Mohit Sharma is the lead, marketing and business development function at PromptCloud. It’s is a Bangalore-based firm dealing with big data solutions such as large-scale web crawling and data extraction. The views expressed are his own and do not reflect those of his employer.

With a petabyte of data being generated every 11 seconds, close to 90 percent of all the data in the world today has been created just in the last two years. With the dawn of user-generated content (UGC), Eric Schmidt opines that every two days we create as much information as we did from the dawn of civilization up until 2003. The data sector is expanding at an unimaginable rate.

Big Data is characterized by the three Vs: volume, variety, and velocity. Volume refers to the amount of data; variety is the number of types that the data has; and velocity refers to the speed of handling and processing data. For the sake of simplicity, big data refers to a collection of chunks of complex information that require specialized software tools to be analyzed. This data is collected in real-time from sources such as RFID sensor data, transaction records, social media profiles and updates, GPS/cellular signal data, etc. World over, organizations have come to realize the importance of this data, though most of them are still perplexed as to how they can incorporate such data-derived insights into their business strategy.

Big data adoption in Asia

When it comes to adoption of ‘Big Data as a Service’ (BDaaS), Asian companies have, for a long time, been fence-sitters, especially when compared with their western counterparts. However, that is changing fast; according to recent research conducted by The Economist’s Intelligence Unit (PDF), a majority of Asian C-level executives feel that technologies such as cloud computing, big data analytics, Software as a Service (SaaS), enterprise mobility, IT security and social media will have the greatest impact on their business strategies in the coming decade.

However, there exists a mismatch between this data point and the actual adoption rate of these technologies by large companies. The research further quotes three reasons for what is holding them back: “a lack of proven success metrics; misalignment between current business processes and the outputs of new solutions; and the diversity of the technology landscape in the Asian market in which they operate.”

Big data in Asia stands at a critical juncture, as there are a few early adopters, but a gap needs to be filled for it to become mainstream. At our level, we feel that Asian executives are quickly grasping the importance of big data and have come to realize the impact that such data can have on their revenues.
With a growing middle class, leading to a rise in consumerism (and e-commerce) in Asia, the number of companies looking to collect and synthesize data regarding consumer behavior and sentiments has also gone up. For instance, we witness requests from our Asian clients to extract geo-specific data that is useful in monitoring of brands, gauging competition behavior, or analyzing consumer sentiments through various social networking sites and blogs. We have perfected our engine to incorporate such queries to extract keyword-based or geo-specific information across most of these platforms, almost in real-time.

Challenges for big data analytics in Asia

Like most upcoming technologies, big data has its own set of challenges to overcome.

  • One major challenge for big data in Asia is that big data has a bigger learning curve associated with it, particularly with respect to use and operations. A very common issue is how one can merge the new data sources with the existing ones, and quite often companies take the easy way out – ‘let’s have both’. This causes integration issues, and also adds to the complexity associated with the use of data.

  • Multi-linguality across the continent poses a significant technology barrier for clients looking to extract web data. At PromptCloud, we have developed our crawl and extraction engine to overcome this challenge – we now daily extract data across 10 to 15 Asian languages.

  • Another hurdle that exists at the operational level for the big data industry is a lack of skilled professionals, having expertise in big data and cloud computing technologies. This talent shortage is set to exacerbate if the growth in the available talent doesn’t match the growth rate of the industry.

(And yes, we're serious about ethics and transparency. More information here.)

Read More