Finding internal data is tough. Here’s how LLMs make it easier

Shreyas Parbat is a former lead product manager at Grab. He uses his data science and software engineering background to build tools for the firm’s data and engineering communities.

Back in 2017, data was deemed a commodity more valuable than oil. The importance of data has only grown since then, with companies across industries storing and utilizing more of it than ever before.

In today’s data-driven companies, every employee is a data consumer who regularly uses in-house data for a variety of tasks, ranging from decision-making via dashboards and reports to training machine learning models by feeding them with historical info.

Image credit: Timmy Loen

This obsession with data comes at a cost. The more data a company stores, the bigger the haystack its employees must search through before finding a dataset that meets their specific needs.

Keyword-based search tools have traditionally been used to find such datasets, but large language models (LLMs) offer a better approach. Here’s how they work and how they can make data discovery easier for various types of businesses.

The data discovery challenge

Streamlining data discovery requires solving problems on multiple levels, from the granular dataset level to organization-wide mental model shifts.

Here are some of the most significant hurdles faced by companies:

Lack of documentation

Let’s face it: Humans are lazy. This behavior may have helped us conserve energy in the wild, but it has also turned into one of the biggest issues in organizations that are larger than a couple of hundred people.

Employees responsible for handling data in a company are usually stretched thin. They create and manage multiple datasets, usually with only their team’s use cases in mind. Unless a dataset is highly popular, adding documentation for it is not prioritized. This makes it very difficult for those not in the know to discover and use datasets.

Reliance on tribal knowledge

The lack of widespread, high-quality documentation limits any search tool’s data discovery capabilities. Naturally, most employees may directly turn to those in data teams for help, usually via internal messaging tools like Slack.

See also: National interests trump AI transparency in Asia

Work smarter, not harder

Data = found