TIA Engineering: On our monorepo move

Photo credit: gmast3r / 123RF Stock Photo.

Over the past five years, the web platform has matured into a robust, performant, and feature-rich runtime that has enabled developers to build truly rich applications right from the browser. Open source frameworks like React, Angular (both 1 and 2), and other languages (Scala.js, Clojurescript, and Elm) are a few great examples that support compilation to JS. These have enabled the modern developer to build beautiful experiences quickly and safely.

As a result, the architecture and size of a front-end JavaScript application have gone from what used to be a couple of files loosely coupled together and facilitated with jQuery, to  hundreds of thousands of SLOC behemoths.

While modern tooling goes a long way in helping to manage this sprawl, the fact remains that, quite unlike languages built for enterprise-level scale, JavaScript was written in a couple of days. Additionally, for a long time, it was solely used to make cute but ultimately trivial effects in the browser.

In fact, the TC39 working committee hasn’t even reached a final decision on how native modules should be handled — a feature that’s par for the course in any language not solely used for trivial purposes.

Some prior art

Tech in Asia’s front end is built on top of React and Redux. Starting off with just a couple of thousand SLOCs, we added the Jobs portal, Business Landing pages, multiple improvements and features to the media/news pages, and the soon-to-be-unveiled revamped Events Landing page. This caused our application to balloon to almost 100,000 SLOCs, with no separation of concerns between these very distinct business domains.

Things eventually reached a tipping point when, in the process of simultaneously developing our events apps for Android and the web, we were required to deploy a stand-alone AWS Lambda function. This was for the express purpose of isomorphically rendering TIA Events-specific markup to be consumed by the Android app.

On the surface, this seemed like a very straightforward task: simply use a tool like Webpack to bundle the app and render the relevant routes as HTML strings with ReactDOMServer.renderToString(). Unfortunately, things are never that simple.

The before shot

Prior to the refactoring that resulted in our FE’s current state, we maintained a very standard, run-of-the-mill project structure — one that you will see reflected in the dozens of React boilerplates in some form or another. It looked something like this:

-- root/
|-- config/
|-- public/
|-- src/
| |-- actions/
| |-- components/
| |-- constants/ (Redux actions)
| |-- middleware/ (Redux store middleware)
| |-- reducers/
| |-- routes/
| | |-- home/
| | |-- article/
| | | ...
| | └-- signup/
| |-- utilities/
| |-- app.js (entry point)
| |-- config.js (pseudo-environment vars etc.)
| |-- routes.js
| └-- store.js
|-- tools (webpack config files etc.)
└-- package.json

While such a directory structure is a perfectly sensible starting point for any React-based app, almost any non-trivial project will outgrow it in very short order. As routes, reducers, and components are added, one ends up with no ability to “pull out” specific components of the app without taking thousands of lines of unnecessary code along with it.

For instance, our new Events page started its life as a directory in the routes folder and eventually morphed into something that may as well have been an app on its own. For various SEO-related reasons, however, it needed to be part of the main app, yet sufficiently decoupled so that it could be deployed and isomorphically rendered without busting AWS Lambda‘s 50mb source size limit. This would have been essentially impossible had we maintained our pre-existing project structure.

Attempt #1: Repo hell

Our first instinct was to extract shared components and major routes into their own git repositories and publish them as private NPM packages. This also seemed like a perfectly sensible idea — and it worked. After all was said and done, the core repository retained its existing directory structure and simply “imported” the necessary packages from its dependent repos. For instance, we started out with an eventslp repo and a core-components repo; all was well and good.

Unfortunately, this still didn’t solve our problem. Many components that the events route depended on used modules that were deeply nested within the core repository. Making it work without multiple Webpack “alias” hacks resulted in multiple repositories, which would then have to be maintained, tagged, merged, and committed to independently — all done entirely by hand.

Attempt #2: Enter the dragon

Why Lerna?