How Tech in Asia achieved 3 engineering milestones in half a year

Photo credits: Pixabay

A journey of discovery

In six months, we grew to a team of seven Engineers, fully migrated our infrastructure to Amazon Web Services (AWS) in August, and attended the first PHPConf.Asia in September.

Along the way, we’ve managed to score 3 major goals. Let’s begin!

Merge Techlist with Tech in Asia

Techlist (now defunct) was a playground of sorts for us. Built on Ruby on Rails, we experimented around with features ranging from mutual introductions to managing watchlists of companies. We allowed investors to scout for promising startups, and founders to scour for suitable VCs. We also crawled the interwebs to pull various site statistics for the companies in our database, and even formulated a score system to rank startups.

The next steps didn’t come easy, however, as it took us a few months to really wrap things up.

Then, we realized we were doing too much. We wanted to focus on what Techlist really was: a startup and investor database. So we rebuilt it from scratch. We took the best of our Techlist knowledge and created models that best represented our data. Unfortunately, we had to bail on a couple of neat features that certain users were accustomed to. We apologize for that. Spoiler alert: return, they will.
​
The next steps didn’t come easy, however, as it took us a few months to really wrap things up. We created scripts to migrate our entire database from MongoDB to MySQL. We then created the backend admin system with Laravel as the base framework, complete with our own code generators and abstractions. It took us multiple iterations to smooth out the kinks for both of these projects.
​
The day we launched our redesign was the day we shuttered Techlist. RIP. Though, it was a welcome change as it would set the tone for our next project… 😉

Full Amazon migration

By August, we were one year into having our site hosted on SoftLayer. Back then, we opted to serve Tech in Asia from the bare metal machines offered. Blazing fast in terms of performance. The trade-off? Reliability and uptime, as we would later realize, in hindsight, and only painstakingly so. In the same period, we experienced 3 major hardware failures that each caused significant downtime while we raced against time to remedy the situation. Hardware-related breakdowns are unfortunately unpredictable, regardless of provider, yet much more impactful. We were thoroughly impressed with SoftLayer’s level of responsiveness and couldn’t find fault with them for these physical realm defects. Still, going cloud-based was a must if we wanted to achieve better availability and uptimes for our users. We chose AWS.
​

We had redundant stacks running for a period of time before finally flipping the switch.

As far as server-side technologies were concerned, everything existed on the physical servers. Hence, we undertook this project in multiple phases. First to go were images. At the time, we were using GlusterFS to synchronize images between our web servers, which coincidentally was beginning to show some not-so-good signs working with half a million files. It took quite a few hours to upload everything onto Amazon’s S3 service. Following that, we were also able to get CloudFront working as our Content Delivery Network (CDN).

Next up were our database and cache services. Setting up RDS (MySQL) and Elasticache (Redis) on AWS were trivial tasks. Maintaining our uptime and data integrity however, took extra effort. By using multiple redundant database instances, we managed to set it up in a way that the switch would appear seamless not only to the user, but to the backend as well.
​
Onto the web server, which is the hardest part. The fact that we were using HHVM (read: fast yet unstable) and that we had to deal with multiple PHP applications on a single instance increased the amount of configuration required. Since HHVM was crashing every now and then, we used a combination of monit and a customized health check application to monitor and restart services as required. To tackle handling multiple services, we introduced a reverse proxy layer in our stack to redirect requests to the right application server. On top of all this, we created recipes for Amazon OpsWorks to configure, setup, and deploy our compute instances, literally allowing us to perform magic with a single click.
​
We had redundant stacks running for a period of time before finally flipping the switch. From experience, it helps to run both in parallel as it gives you time to find breaking issues and fix them. It also reduces your stress levels knowing that you’re able to revert in case of an emergency.
​
Today, we are very lucky to have an additional production-mirrored environment for QA purposes, as well as an automatic build-and-deploy process powered by Jenkins. We’re also able to accomodate sudden traffic spikes by spinning up additional instances from our stack. We sincerely hope your experience on Tech in Asia reflects these infrastructure upgrades.

Microservices architecture

2016 and beyond