TIA Engineering: How we’re building our data infrastructure

Photo credit: John Voo.

At Tech in Asia, we’re constantly striving to deliver a better experience to our users through our conferences, events, apps, and content. We have unique offline and online experiences which are challenging to consolidate. And as a content platform, content for us is key.

Since data is fundamental in improving user experience, we have to cohesively consolidate all our touch points with our users so that the business can understand and use data in decision making.

Having rock solid data infrastructure is the first step toward making us better organizers, writers, and creators.

Design principles and philosophy

In setting up our infrastructure, we follow several guiding principles based on the constraints we have and the experiences we have gleaned from others.

• Small team: We wanted to leverage managed services in order to minimize operational overhead, as this reduces the time we have to do cool things.
• Correct tools for the problem at hand: No tool is a silver bullet. There are constraints and trade-offs with each choice.
• Flexibility: We wanted to enable as many tools as possible without too much diversity in the skill sets that we are building for.
• Scalable: We might not have a lot of data now but we want to build in a moderate level of redundancy, just in case.

Component overview

Version 1 of our data infrastructure.

Our data stack is currently entirely based on Amazon Web Services (AWS). It offers several managed services that we can use instead of rolling our own servers, which our engineers would then have to maintain. For example, many companies use Kafka as their Pub/Sub and events collection service, but we use Kinesis Firehose instead.

While Kafka provides a richer set of capabilities than Kinesis Firehose, maintaining it is an issue. We collect around 1 million events daily, which, relative to many companies, is not a lot. This makes Kinesis Firehose a more viable solution as compared to rolling our own Kakfa cluster. Furthermore, we have limited use cases for real-time data. We only process the data hourly and it has therefore satisfied many of our needs.

At the application layer, we run MySQL servers for our media, events, jobs, and startups data. They are the primary data storages for the APIs that serve our web and mobile front-ends. For more specialized needs such as aggregation and analytics, we use PostgreSQL .

In that respect, we have two primary data repositories, one for non-volatile data, such as application logs and raw data (Data Lake), and the other for business reporting purposes (Data Warehouse). Our Data Lake is based on Hive, with its underlying data stored in Amazon S3, while our Data Warehouse is based on Redshift. Instead of putting all our data in Redshift, we chose to segregate it due to some of our use cases and economic reasons, as storing hundreds of GBs or TBs of data in Redshift is costly.

However, allowing access to our data across Redshift and Hive is of paramount importance. So, we have Presto, which allows data from multiple databases to appear as if they are in one database. With it, we can also use events data stored in Redshift in conjunction with user data stored in MySQL and even logs from Hive.

To process the data, we use Airflow as our scheduler and coordinator. Airflow is a workflow management software released by Airbnb, which allows one to run SQL, Shell scripts, Python code, etc. at a predetermined schedule. These tasks and their dependencies are in the form of a Directed Acyclic Graph. We run our ELT/ETL process with Airflow, which consists of a myriad of tasks like running Sqoop, SQL, Spark applications, Presto queries, Python scripts, and Shell commands that perform quality checks and data extraction, transformation, and validation together.

What’s next?