Your guide to how ChatGPT actually works

The internet is awash with excitement and anxiety about ChatGPT, but I haven’t seen many explainers on how it actually works. There are technical papers out there, but they can be daunting for the layperson.

This article attempts to demystify the chatbot and some of the jargon surrounding it. First things first, let’s start with the basics.

ChatGPT is a chatbot, a computer program that can simulate conversations with human users. Chatbots use natural language processing (NLP) to understand what the user types and then generate a relevant response.

ChatGPT uses Generative Pre-trained Transformer (GPT), a large language model created by OpenAI. Besides ChatGPT, there are other similar chatbots, including Bard by Google and Claude by Anthropic.

Large language models

Large language models (LLMs) are AI models that learn and generate human language. LLMs are a core part of NLP and include GPT by OpenAI and Language Model for Dialogue Applications (LaMDA) by Google.

These models learn from a large amount of text data and then use what they’ve learned to generate or understand new text. Basically, LLMs predict the next word in a sequence using the words that came before it.

For example, if you have this sequence of words:

The quick brown fox jumps over the lazy

The LLM will predict the next word:

The quick brown fox jumps over the lazy dog

The model does this using an AI technique called machine learning.

Machine learning

Machine learning is a family of AI algorithms that involves ingesting large amounts of data, which is then used to train an AI model to make decisions.

Let’s say you teach a child what a dog looks like by showing them pictures of different dogs. After seeing enough pictures, the child starts to understand the general characteristics of a dog: It has four legs and a tail; its size varies, but there’s a common facial structure; and so on. When a child finally sees it in real life, they can tell it’s a dog because of the patterns they’ve learned.

Neural network

Transformers

Tokens

How ChatGPT learns

Fine-tuning

Different types of models

ChatGPT training