A language model predicts the next token. This video follows that prediction through the generation loop to a complete chat response. We'll watch the generation loop run, see why two runs of the same prompt can come out differently and which knobs control that, and see how chat is really completion over a transcript. Then we look at what every call to the model actually carries, and at the context window, the fixed amount of text the model can see at once.
Let's watch the generation loop run, step by step. We start with our context already in place. The cat sat on the.
The model predicts a distribution over the next token, just as it did before. Mat carries the highest probability here, so it is the most likely next token.
But here is something important. We don't always take the most likely token. Instead we sample. We pick a token at random, but weighted by the probabilities, so more likely tokens are chosen more often. This time the choice landed on hat, even though mat was more probable. That randomness is deliberate, and we'll see in a moment why it's useful.
We append the token we picked, hat, and from now on it behaves exactly like any other token in the context.
Now the loop repeats. With hat added to the context, the model predicts the next token all over again. It might be a period, or the token representing the word "and", or the word "today".
We sample once more. This time the sample is a period, and we append it, which ends the sentence.
That is the whole engine. Predict a distribution, sample a token, append it, and repeat. Because there is sampling at each step, the very same prompt can produce different text on different runs.
Let's make that randomness concrete. We give the model the exact same prompt every time. The cat sat on the.
Run it once and we get one continuation. The cat sat on the mat, the cat slept.
Run the identical prompt again, and we can get a completely different ending.
A third run gives another continuation. The input and settings are the same, but the sampled tokens differ.
Sampling gives us variety, but it also makes comparison harder. The same prompt can produce different answers, so one run is not enough to judge how reliably it works. That variety is useful when we want several possible names, explanations, or approaches to a problem.
Since sampling is what drives the randomness, we get a few controls over how random it is. The main one is called temperature, and it reshapes the distribution before we sample from it. At a temperature of one, we use the model's probabilities as they are, unchanged.
Turn the temperature down, and the distribution becomes sharper. The model leans heavily toward its top choice, so its output is more predictable. Turn the temperature up, and the distribution becomes flatter. Less likely tokens have a greater chance of being chosen, so the output can vary more. More variety does not necessarily mean a better answer.
Temperature isn't the only control we have. Top-k keeps a chosen number of the highest-probability tokens. Top-p keeps enough of them to reach a chosen share of the total probability. Both narrow the set we sample from. Other controls include a limit on output tokens and, in some APIs, a random seed to help repeat a run. A seed is not always a guarantee of identical output. By the way, this is because the GPU does many of these calculations in parallel, and the order in which it adds the numbers up can change from one run to the next. The result differs in the last few digits, and that is sometimes enough to change which token is chosen. We'll meet these controls again when we make model calls ourselves.
Here is the mental model to keep. Temperature reshapes the distribution, top-p and top-k trim away its tail, and then we sample a token.
How can next-token prediction produce useful abilities like arithmetic or translation? To see why, look at what predicting the next token actually requires. Consider the sentence, if three pens cost six dollars, one pen costs. To answer this kind of question reliably, the model needs to use the relationship between the numbers. Here the answer is two.
For the example English cat, French, the correct continuation is chat. Learning relationships between languages helps the model complete this kind of text.
For the capital of Japan is, the expected continuation is Tokyo. Facts learned during training help the model predict that answer.
These tasks can all appear as text for the model to predict. Learning arithmetic, translation, and facts about the world helps the model predict these different kinds of text. The same next-token training goal gives it a reason to learn each of those relationships. So several distinct skills can develop from this one training goal. We can then test how well the model uses each skill on new problems.
So far our model only completes text. How do we get a chat assistant out of it, the kind of thing ChatGPT made famous? The idea is simpler than it might seem. We write the whole conversation out as one long piece of text, with small role markers, like user and assistant, tagging who said what.
That entire transcript, every turn of the conversation so far, is fed in as one single sequence of tokens. There is no separate chat engine underneath. It is the same next-token predictor, now reading the whole dialogue.
To answer, the model simply completes the text right after the latest assistant marker. Here it predicts it's Tokyo, one token at a time, exactly the way it would finish any other sentence.
But the model also has to know when to stop. During training it learns a special end-of-message token, and it emits that token once the reply is complete. When the stop token appears, this reply ends. In an ordinary chat, the interface then waits for the user's next message.
The basic mechanism is completion over a transcript with role markers. The same predictor we've been building reads the full conversation and completes the assistant's reply. The role markers give the conversation its structure. Training also teaches the model how to respond as an assistant.
This reframing has a consequence that is easy to miss, so let's look at exactly what gets sent to the model each time the harness makes a request. Here is the first request. It holds the system prompt, which is the harness's own instructions to the model, and after it your task, tagged as the user's turn. The model reads all of it and completes the next assistant turn. In this example, that turn is a tool call asking to read the test file.
Now look at the second request. It is not just the new material. The system prompt and your task are sent again, word for word, and only after them come the model's tool call and the result the harness read back, each tagged with its role. The model keeps no separate conversation history to fill in what a request leaves out. The harness sends the whole transcript so far again, every time, as these requests show.
By the third request the pattern is clear. Only the last two lines are new, and everything above them is the same text, sent for the third time. This is the point to hold on to. Each request carries the whole current window, not just the line you added. The repeated part grows with every turn until the harness summarizes or removes some of it.
Sending the same text again and again sounds wasteful, and the provider does something about it. The model runs in the cloud, and the work it did on a beginning it has already seen is kept for a short while and reused. This is called prompt caching. When a new request starts with a matching cached prefix, the provider reuses its work on that part and processes the new input. The repeated text is still in the request, but reusing the computation reduces the wait and the cost of processing it.
The cache only matches a prefix that has not changed. If anything earlier in the request is different, a different system prompt, an edited earlier message, a different set of tools, then from that point on nothing matches, and the model processes the rest of the request from scratch. The cache can also expire. How long it lasts depends on the provider and settings. Without a usable cache, the model has to process the input again before generating the answer.
Now, remember that the entire conversation is fed in as one long sequence of tokens. As the chat continues, that sequence only keeps growing.
But a model can only attend to a fixed number of tokens at once. That limit is called the context window. Longer sequences take more memory and computation. In the full attention pattern shown earlier, each position can use all the positions before it, so the number of comparisons grows quickly.
Everything the model uses to answer, the instructions, the earlier turns, and any files you have shared, all of it has to fit inside that window. The reply also needs room. As the request approaches the limit, the harness has to decide what to keep, summarize, or remove.
Let's recap. The model represents text as tokens and learned vectors, then predicts a distribution for the next token.
Generation is a loop that samples one token from that prediction, appends it, and predicts again, which is why two runs of the same prompt can differ.
Chat is just completion over a transcript tagged with roles, read within a finite context window.
This gives us a way to understand the agent's replies: what information was in the window, and what did the model generate from it? We'll use that distinction when we compare runs.