Data engineer vs data analyst vs data scientist, explained

Photo credit: Dmitriy Shironosov

I work at a data science mentorship startup called SharpestMinds and I’ve found that there’s one advice I give over and over: that aspiring mentees first think about what kind of data scientist they want to be.

This is crucial because data science isn’t a single, well-defined field. Companies don’t hire generic, jack-of-all-trade “data scientists,” but individuals with very specialized skill sets.

When a company is hiring for a data scientist, they often have a well-defined problem in mind that will require specific technical know-how and subject matter expertise.

For example, some companies apply simple models to large datasets, some apply complex models to small ones, some need to train their models on the fly, and some don’t use (conventional) models at all.

Each calls for a completely different skill set, so it’s especially odd that aspiring data scientists receive generic advice (learn how to use Python, build some classification/regression/clustering projects, and start applying for jobs).

Those of us in the industry bear the blame. We tend to lump an excessive number of things into the “data science” bucket in casual conversations, blog posts, and presentations. Building a robust data pipeline for production? Inventing a new kind of neural network? That’s a data science problem.

That’s not good because it can cause aspiring data scientists to lose focus on specific problem classes and instead become jacks of all trades . This can make it harder to get noticed in a market that’s already saturated with generalists.

But it’s hard to avoid becoming a generalist if you don’t know which common problem classes you could specialize in. That’s why I’ve put together this list of five problem classes that are often lumped together under the data science heading.

1. Data engineer

Job description: You’ll be managing data pipelines for companies that deal with large volumes of data. That means making sure that your data is being efficiently collected and retrieved from its source when needed, cleaned, and pre-processed.

Why it’s important: If you’ve only ever worked with relatively small datasets (less than 5 GB) stored in .csv or .txt files, it might be hard to understand why there would exist people whose full-time jobs it is to build and maintain data pipelines.

Here are a couple of reasons:

1. A 50 GB dataset won’t fit in your computer’s RAM, so you generally need other ways to feed it into your model
2. Much data can take a ridiculous amount of time to process and often has to be stored redundantly. Managing that storage takes specialized technical know-how.

Requirements: The technologies you’ll be working with include Apache Spark, Hadoop and/or Hive, as well as Kafka. You’ll most likely need to have a solid foundation in SQL.

2. Data analyst

3. Data scientist

4. Machine learning engineer

5. Machine learning researcher