What to do when there’s too much data?

Data as a business resource has become so valuable that it’s been dubbed “the new oil.” High-profile examples such as Uber’s dominance in the transportation industry illustrate how taking a data-driven approach can help startups catch up quickly to – and even overtake – large, entrenched incumbents.

And with businesses increasing their digital transformation efforts in Southeast Asia, they’re grappling with more consumer data than ever before.

“Every organization has a wealth of data about its own business operations,” says Mark Van de Wiel, vice president of technology at data solutions firm HVR, which was acquired by US-based Fivetran in September. “Whether that’s how they recruit customers or how the customers interact with the products, it’s become more and more clear that data is a differentiator.”

Mark Van de Wiel, vice president of technology at HVR / Photo credit: HVR

However, if businesses don’t handle and refine data properly, they won’t be able to make the most of its value.

It’s not that simple

While data management and analytics tools abound in the market, data comes from and is siloed in many different locations. The need to bring the information together has traditionally been one of the challenges that DataOps professionals face. DataOps (short for data operations) is a methodology that brings together DevOps and data teams to support the development and delivery of analytics.

For HVR’s clients, who are often in the locomotive manufacturing industry, a lot of work goes into determining when, where, and how to best execute maintenance services for physical equipment.

“The locomotive moves around, it’s not static, and it’s a complex piece of equipment. So the company needs to get a good engineer to the right place at the right time with the right parts to service the locomotive,” Van de Wiel says.

In order to do this, the company uses internet of things sensors that are built into the machines to track measurement changes over time, geolocation data, as well as weather data – among a wealth of other retrievable information.

Additionally, with Covid-19 sparking a sudden and significant shift in companies going digital, there has also been an explosion in the sheer quantity of data produced and collected. This presents its own difficulties, with the systems of different businesses unprepared to handle this new load. “Those systems have become busier, heavier-loaded, and also they are now active 24/7,” Van de Wiel says.

The large volume of data and ensuing runtime challenges also make it harder to run analytics processes and scenarios. Under these circumstances, it’s difficult to get the level of performance required to extract business value from the data efficiently and effectively, particularly with the complex algorithms that predictive and prescriptive analytics use.

“How can we get to the data that sits in the system knowing that the system is continuously busy?” Van de Wiel ventures.

Best practices

A lot of the most mission-critical data for organizations today resides in relational databases – a type of database that stores (and provides access to) data points that are related to one another – and that’s likely not changing anytime soon.

With that in mind, and with so many data processing systems running at such high volumes, HVR’s answer to acquiring data without causing disruptions is its log-based change data capture feature.

This technology captures data changes in the databases in real time and replicates it, moving only the changed data downstream to other use cases and the analytics systems. In this way, there is minimal impact on the performance of the source systems, allowing companies to run scalable levels of analytics without disrupting ongoing processes.

Log-based change data capture is part of HVR 6.0, the firm’s latest data replication solution that efficiently integrates large data volumes in complex environments. Besides enabling more efficient data analytics, it also makes it easy for firms to automate data gathering routines while also providing data validation methods.

Photo credit: HVR

Validating information is particularly important for companies in the finance industry. “[These firms] have to know that the data is accurate in order to be compliant with laws and regulations,” Van de Wiel explains. “If they no longer take the data directly from the source system, they need to know that the data is in fact accurate.”

Tomorrow’s hurdles

Looking ahead, Van de Wiel sees one key development that will have far-reaching implications on the DataOps scene: the proliferation and increased adoption of software-as-a-service (SaaS) solutions.

Rising SaaS adoption is borne out of organizations’ desire to focus more key resources on their products and customers and less on managing back-end functions. “As an organization, you want to focus on what makes you more competitive in your market, and managing software components – while necessary – is unattractive and just not a part of that,” he says.

This means that information technology (IT) departments across the board are gradually shrinking. Instead, revenue-generating parts of organizations are receiving IT budgets to drive faster and better outcomes. For HVR, this translates to a shift in their target audience, which now includes less tech-savvy users.

For its part, the company is working on making its software available as a service in anticipation of this development.

As a final piece of advice for companies, Van de Wiel recommends that they do their best to limit the number of vendors they work with. Choosing the right vendor that can help to solve as many problems as possible will lower costs and reduce the number of moving parts, cutting down on complications and putting the emphasis on efficiency.

Working with too many vendors brings up another major concern: security. “That’s where you have to be very careful that you pick one that not only has strong security capabilities, but also one that understands the right implementation of the service – it’s not always a given that the vendor has the appropriate security protocols in place,” he concludes.

HVR’s real-time data replication technology enables organizations to plan, predict, and respond with the freshest data available. Used by leading global organizations, HVR’s solutions help its clients to better serve customers, reduce margins, plan resources, and ultimately improve their bottom line.

Find out more and take a test drive on HVR’s website.

This content was produced by Tech in Asia Studios, which connects brands with Asia’s tech community. Learn more about partnering with Tech in Asia Studios.