The Engine of Conversation: How ChatGPT Actually Works

Step 1: The Prompt Becomes Tokens

The model does not see words or sentences. It sees 'tokens'. A token is a common sequence of characters. For English text, one token is roughly equivalent to four characters or three-quarters of a word. The prompt “How does a transformer model work?” might be broken into tokens like `["How", " does", " a", " transform", "er", " model", " work", "?"]`.

This tokenization process converts the human-readable prompt into a sequence of numbers that the model can process mathematically. The entire vocabulary of the model is a massive list of these tokens.

Step 2: The Transformer and 'Attention'

The numerical sequence of tokens is fed into the model's core architecture: the Transformer. First detailed in a seminal 2017 paper by researchers at Google, "Attention Is All You Need," the Transformer was a breakthrough in processing sequential data like language. Its key innovation is the 'attention mechanism'.

Instead of processing text sequentially like older models, the attention mechanism allows the model to dynamically weigh the importance of every token in the input relative to every other token. When generating a word, it can 'pay attention' to another word several sentences back that provides crucial context. This ability to build a rich, contextual understanding of meaning, nuance, and relationships across long passages of text is what gives models like ChatGPT their power and coherence.

Step 3: Predicting the Next Token

With the context of the prompt established, the model performs its fundamental task: it predicts the single most likely next token. It calculates a probability score for every token in its vast vocabulary. It is a massive statistical guess.

A parameter called 'temperature' influences this guess. A low temperature (e.g., 0.2) makes the model more deterministic; it will almost always pick the token with the highest probability, leading to focused, conservative answers. A high temperature (e.g., 0.9) increases randomness, allowing the model to pick less likely tokens, which can result in more creative or diverse, but also less coherent, output.

Once a token is chosen, it is appended to the input sequence. The entire process repeats. The model now considers the original prompt plus the first token it generated, and predicts the second. Then the third. Then the fourth. This token-by-token generation continues until the model produces a special 'stop' token or reaches its maximum output length. The full sequence of generated tokens is then converted back into words and displayed to the user.

Phase 1: Pre-training on the Internet's Library

The first phase is 'pre-training'. The model is shown a gargantuan dataset of text and code, scraped from the public internet. This includes an enormous volume of books, articles, websites, and scientific papers—hundreds of billions of words. During this phase, the model's sole objective is to learn to predict the next word in a sentence. By doing this billions of times over, it implicitly learns grammar, facts, reasoning styles, and the myriad patterns of human language.

This stage is unsupervised. The model learns these patterns on its own, forming a powerful but raw foundation of knowledge. The result is a 'base model' that can complete text but is not yet a helpful, conversational assistant.

Phase 2: Fine-Tuning for Conversation and Safety

A raw base model is not safe or useful for public use. It might produce harmful, biased, or nonsensical text because it only reflects the patterns in its data, good and bad. The next phases aim to 'align' the model with human values and conversational expectations.

First is Supervised Fine-Tuning (SFT). Human AI trainers create a high-quality dataset of conversations where they play both the user and the AI assistant. They write the prompts and the ideal responses. The base model is trained on this curated dataset to learn the conversational question-and-answer format.

The second, more complex stage is Reinforcement Learning from Human Feedback (RLHF). Here, the model generates several different answers to a single prompt. Human trainers then rank these answers from best to worst. This ranking data is used to train a separate 'reward model'. The reward model's job is to look at any response and predict how a human would rate it. Finally, the main ChatGPT model is further fine-tuned using the reward model as a guide, optimizing its own responses to achieve the highest possible 'reward' score. This process teaches the model to be more helpful, to follow instructions, and to refuse inappropriate requests.

The Problem of 'Hallucinations'

Because the model's goal is to generate statistically plausible text, not to state verified facts, it can and does invent information. These fabrications, often delivered with the same confident tone as factual statements, are known as 'hallucinations'. The model might invent academic citations, make up historical events, or create biographical details that are completely false. It does this because the invented text is a statistically likely sequence of tokens, not because it is actively trying to deceive. It is a pattern-matcher, not a fact-checker.

The Context Window and Memory

A single conversation with ChatGPT does not have infinite memory. The model can only 'see' a certain amount of text at one time, a limit known as the 'context window'. This window includes your prompt, its response, and the recent history of the chat, up to a fixed number of tokens (e.g., a few thousand to over one hundred thousand, depending on the model version). Any part of the conversation that scrolls past this window is forgotten. It cannot refer back to the beginning of a very long chat.

Furthermore, the model has no inherent memory between separate conversations. Each new chat session starts from a blank slate, with no knowledge of who you are or what you have discussed before. Features that provide long-term memory are separate additions that work by saving and re-injecting information into the prompt for new sessions.

A Fixed Knowledge Base

The core model's knowledge is not live. It is frozen at the time its pre-training data was collected. For many versions of ChatGPT, this means it has no knowledge of events, discoveries, or data that emerged after its 'knowledge cutoff' date, which is often a year or more in the past. While some versions can access the internet to retrieve current information, this is a separate tool-use capability, not a feature of the underlying base model itself.