A founder’s guide to navigating the “trough of sorrow”

While I was reading some success stories on Indie Hackers, it occurred to me that my project, Get Data, really took longer than others to gain significant traction – a full five years to be exact.

The beginning

I first stumbled upon this project in December 2012 when I was trying to solve two problems I personally encountered.

In my first problem, I was trying to identify the best stocks to buy on the Singapore Stock Exchange. While browsing through the stocks listed on the website, I soon realized that most stock exchanges and other financial websites gear their data presentation towards quick buy and sell behaviors.

If you are looking to get data for granular analysis based on historical company performance as opposed to stock price movements, you’d notice that it’s like pulling teeth. Even then, the important financial data I needed for decision-making purposes were spread across multiple websites.

This first problem led me to write two web scrapers – one for SGX.com and the other for Yahoo Finance – to extract datasets, which I later combined to help me with my investment decision-making.

Once I happily parked my cash, I went back to working on my side project. It was a travel portal that aggregates travel packages from agencies across Southeast Asia.

Then, I encountered my second problem. I had to write a bunch of web scrapers again to pull data from vendor sites, which did not have APIs! Being forced to write my third, fourth, and maybe fifth web scraper within a single week led me to put on hold all work and step back to look at the bigger picture.

The insight

Being a web developer and understanding how other web developers think, it quickly occurred to me the patterns that repeat themselves across webpage listings and nested webpages. This is especially true for naming conventions when it comes to CSS styling. Developers tend to name their CSS classes the way they would actual physical objects in the real world.

I figured that if there existed a semantic query language that is program-independent, it would provide the benefit of querying webpages as if they were database tables while providing for clean abstraction of schema from the underlying technology. These two insights still hold true today after six years into the project.

While the first five years depicted in the graph above seem peaceful due to a lack of activity, it felt anything but peaceful. During this time, I was privately struggling with a bunch of challenges.

Team management mistakes and premature scaling

The first challenge was team management. During the inception of the project, my ex-schoolmate approached me to ask if there was any project that he could get involved in. I naturally invited him to join me in my current project and soon got ourselves into an incubator in Singapore called JFDI.

In hindsight, while the experience provided us with general knowledge and friends, it really felt like going through a whirlwind. The most important piece of knowledge I came across was a book recommendation: The Founder’s Dilemmas. I wished I read it before I made all the mistakes I did.

First signs of life

Technical challenges

Traction and growth

Continued education

Moving forward