Behind the scenes of achieving data speed and accuracy

For companies, the importance of moving fast in today’s digital world has long been established. Now, the key lies in moving not just quickly but also with certainty.

Making decisions swiftly means that businesses are more reliant on data than ever before. However, the sheer volume of data being generated today – around 2.5 quintillion bytes of data each day – makes it difficult for companies to make those decisions.

Aaron Katz, CEO and co-founder at ClickHouse / Photo credit: ClickHouse

“The trend that we’ve observed over the past few years is the significant increase in the volume of data being generated, and the need to analyze that data hasn’t changed,” says Aaron Katz, co-founder and CEO of ClickHouse, a high-performance database built for real-time analytical workloads.

Keeping it real

As demand for data analytics has grown, a number of trends have emerged such as object storage technology, which is offered by the likes of Amazon and Google Cloud. While these solutions offer efficiency and cost savings, having data distributed across different platforms makes it difficult for a company to properly utilize them.

Photo credit: wutzkohphoto / Shutterstock

That’s why gaining access to real-time analytics has become a non-negotiable for businesses.

Real-time analytics helps companies understand what’s going on inside their tech stacks. It also enables them to respond better and faster to changes in the business landscape.

“With data streaming in real time, you’ve got tens of thousands of concurrent users querying that data,” Katz explains. “There’s an expectation for you to respond in less than 100 milliseconds, if not faster. And to do that, you need extremely quick performance over trillions of records or events.”

The power of observability

These requirements, coupled with the recent explosion of data, have helped fuel the emergence of firms such as ClickHouse, which give companies access to infrastructure that powers faster data analytics.

These firms are focused on boosting companies’ “observability.” According to Katz, observability enables businesses to quickly pinpoint the root causes of an issue, regardless of how complex their tech stacks are.

In other words, the more observable a system, the better the business’ decision-making.

For example, a ride-hailing company aims to use multiple data points – such as GPS location or the type of operating system that a customer has – to inform its decisions in situations such as surge pricing schemes.

Photo credit: Kaspars Grinvalds / 123RF

“Maybe your demand exceeds supply in a certain neighborhood. You’re going to have to make a surge pricing calculation in less than a second,” Katz points out.

In these scenarios, even minor delays in data queries could lead to poor user experiences.

Demand for real-time analytics will only intensify amid accelerating adoption of AI solutions, which depend on continuous streams of information to power decision-making for autonomous vehicles, robots, and other similar technologies.

“Think about Tesla’s self-driving cars, which generate a billion events per second that have to be ingested, analyzed, and stored efficiently. That’s a very challenging engineering feat,” adds Katz.

The ClickHouse difference

ClickHouse was initially developed by tech firm Yandex in 2009 with the goal of being the fastest online analytical processing database in the world.

“Our technology ingests a company’s data and helps it process it to get an output very efficiently and quickly,” explains Katz.

In 2020, ClickHouse helped ecommerce firm Shopee handle its rapidly growing platform during the Covid-19 pandemic. Amid surging daily transactions, Shopee’s engineers faced mounting user complaints about disruptions and slow service, and the team’s ability to understand where problems were occurring were stymied by the system’s lack of observability.

Photo credit: Shopee

ClickHouse enabled Shopee to adopt a technique called “distributed tracing” to pinpoint its performance bottlenecks and errors as well as implement fixes in a timely manner.

“Every minute of downtime is lost revenue, so the minute you have an issue, you need to be able to notify someone,” says Katz. “Within three seconds, you need to have already increased your performance load or provision more resources to bring your system back up.”

Electrifying data

Effective observability isn’t simply focused on reacting to problems as they arise. It also means taking preemptive actions.

That’s why automation and AI features were designed and integrated into ClickHouse’s database management solution, says Katz.

He brings up ecommerce as one use case. “The system is learning from data flows over time to understand normal patterns of behavior, so if you’re buying something with a credit card, a bunch of different checks have to be made to ensure the transaction isn’t fraudulent. That decision tree has to occur in less than a second to determine if the vendor or processor will accept the payment.”

ClickHouse is also helping businesses scale their operations. In Indonesia, e-bike firm Electrum manages a fleet of over 3,000 vehicles. With the business’ rapid expansion, its engineering team was suddenly faced with a large amount of data demands from vehicles, customer transactions, and fleet operations.

Photo credit: Electrum

To meet these demands, Electrum integrated ClickHouse’s data management solution, whose columnar storage design allows for much faster SQL queries, into its operations. This lets the startup’s system generate analytics quickly.

With the integration, Electrum is able to process over 500 million records daily. And despite this improvement, the startup only spends US$500 a month on its server setups.

The future is open source

While much has changed for ClickHouse over the years, it has remained committed to its open-source roots, which Katz says is essential to its ability to innovate.

“You get your technology in the hands of millions of customers, and that broader user base contributes to its development,” he adds.

As AI adoption shapes how businesses think about their technology investments in the coming years, the open-source approach will only become more important.

“Almost every product development decision in almost every industry is thinking about how they can implement AI,” Katz points out. “With open-source AI technologies, you can apply models to your proprietary and sensitive data within your own environment without having to share that confidential information with a large language model supplier.”

An open-source approach could give companies an advantage when it comes to managing the governance risks that tend to arise around AI. At the same time, blending AI with an agile database solution could enable companies to respond faster to any future changes in regulation or tech trends- – keeping them ahead of the curve as tech marches on.

ClickHouse is a part of the IMDA Accreditation program, which was launched in July 2014 to accelerate the growth of promising Singapore-based enterprise tech firms. The program also aims to help such companies establish their credentials, build business traction, compete in the global market, and gain more opportunities to showcase their solutions to spur adoption.

LEARN MORE

ClickHouse is the fastest and most resource-efficient real-time database and data warehouse, optimized for diverse data-intensive workloads. Learn more about it on its website.

This content was produced by Tech in Asia Studios, which connects brands with Asia’s tech community. Learn more about partnering with Tech in Asia Studios.