So far we have worked with the agent as a whole. In this video we look at the language model inside it. We'll start with the division of work between the harness on your computer and the model on a remote server. Then we'll focus on the model. We'll see how text becomes tokens, how each token becomes a vector of numbers, how the model predicts one token at a time, and what the stack of layers inside it looks like. Each of those explains something you will notice in the agent's behaviour later on. We use these models in this course rather than build them, so we'll stay at the level of intuition throughout.
Let's begin by recalling the shape of an agent, because it tells us exactly where the model sits. It begins with you. You hand a task, described in plain language, to the harness. The harness is the program running on your own machine, the one you installed and are typing into.
The harness gathers your task and relevant information about the project, then sends a request to the model over the network. In this setup, the model runs on a remote server. This is worth being precise about. The model is not on your machine, and it cannot see your machine. Everything it knows, the harness had to send it.
The model considers the request, and replies. That reply is often not an answer for you to read. Instead it is a request of its own, asking for a tool call. A tool is something concrete that can be done in your project, like reading a file, editing code, or running a command. The model can ask for one, but it cannot carry it out.
The harness handles the tool call under the permissions in place. In this example, the tool runs on your computer, and the harness collects its result. That might be the contents of a file, the output of a test, or a confirmation that an edit was made.
The harness sends that result back to the model in the next request, and the model decides what to do next. The model can ask for another tool call, receive its result, and choose another step. This is the agent loop. A step can help with the task, but it can also fail or need correcting.
When the model returns a final reply instead of another tool call, the harness hands the result back to you. This is when the text from the agent's answer finally shows up in your chat. The model proposes actions. The harness controls which tools are available, handles permissions, and runs the allowed calls. For the rest of this video, we'll look at how the model produces those replies and tool calls.
A model never sees your text the way you do. The first thing that happens is that your raw text gets broken up into pieces called tokens. Here is the phrase, tokenization is simple, before we split it.
Here is the result of the split. Notice that the word tokenization became two tokens, token and ization, while the word simple stayed whole. So tokens are not the same as words. They are chunks of text, and common words tend to be a single chunk, while rarer words get broken into several. You can even see that some tokens in our example include the space character that separates words in text.
A tokenizer has a vocabulary: the set of tokens it uses to represent text. For concrete numbers throughout this video we'll use Llama 3, because it was released openly, so its sizes and design are publicly documented. Its vocabulary holds around a hundred and twenty-eight thousand tokens. The model works with tokens. A token can represent a whole word, part of a word, punctuation, or other pieces of text.
Tokens help explain why counting letters can be awkward for a language model, even when the question looks simple to us. How many times does the letter r appear in the word strawberry? A familiar wrong answer from language models is two. The correct answer is three. Newer models usually get this particular word right. The question became so famous that it now appears in their training data, so a correct answer may be memorized rather than worked out.
When you read the word, you see ten separate letters laid out in front of you. You can point at each r in turn and count them, and you get three.
The model does not receive the word that way. This diagram shows the case where strawberry with a leading space is one token. The exact split depends on the tokenizer. The model receives a token identifier for the whole word, rather than a separate identifier for each letter. It has to work out the spelling before it can count the letters.
In some language models, the word strawberry might be split differently. Splitting the word into a few tokens still does not give the model one token per letter. The pieces are still chunks of several letters at a time, and the model works on the chunks rather than on the letters inside them. The model can learn how tokens are spelled during training. But that is different from receiving a list of letters it can count directly.
How can we make a counting task more reliable? One approach is to put the letters into the text itself. A model that first writes the word out one letter at a time then has something it can actually count. Writing out the letters can help, but it does not guarantee the count. The model may spell the word correctly and still count it incorrectly. For an exact count, the agent can run a line of code. We can inspect the string and the counting operation, then check the result. That is a useful role for tools: a model can ask a program to carry out an exact operation. Knowing how the text is represented helps us choose a better way to do the task.
How are tokens used in the computation? Each token has a numeric identifier, but the network needs more than that identifier to work with. It uses a learned representation of the token. The model stores these learned representations in a lookup table, with one row for every token in the vocabulary.
To turn a token into numbers, we simply look up its row in that table. Here we find the row that belongs to ization and pull it out.
That row is a list of numbers, a vector, and this vector is called the token's embedding. There are a great many numbers per token. In Llama 3's largest model, each embedding vector holds roughly sixteen thousand numbers.
The values in an embedding come from training. The table may start with random values, but training adjusts them along with the network's other parameters. Training can place tokens with similar meanings or uses near each other in this vector space. Distances and directions then represent learned relationships between tokens. This is not always the case, but it hints at how language models encode information inside them.
So what does the model actually do with all of those vectors? For text generation, the model predicts what token could come next. Our example starts with the words, the cat sat on the.
We feed that text through the model. For now let's treat the model itself as a black box. We'll open it up in a moment.
What comes out is not a single answer. It is a probability for every possible next token in the vocabulary. In this illustration, mat has the highest probability at forty percent, followed by chair, hat, and rug. We are showing a few candidates from a distribution over the whole vocabulary. To generate text, we choose a token from this distribution, append it, and ask for the next prediction. Repeating that process produces the response.
Let's open up that black box, at the level of intuition, since this course *uses* these models rather than constructing them rigorously. We start with our input tokens laid out left to right. The cat sat on the hat.
Each token becomes its embedding vector, the same numbers we just looked up in the table. So now every token position has its own vector of numbers sitting above its word.
Above each token we build a tall stack of layers, with the same structure repeated. There are two kinds of layer, and they alternate. Attention layers are shown in one color, and feed-forward layers are drawn as the small network glyphs in the other. Real models stack many of these layers. Here we're showing just two of each, to keep the picture readable.
The part that combines information across token positions is called an attention head. The feed-forward layers process each position on its own. The attention layers are different. They let a token look back at the earlier tokens and pull in information from them. That is how the model takes the surrounding context into account. We won't explain attention heads here in more detail. The point is that each position can use information from the tokens up to that position.
At the very top of the last stack, the model produces exactly what we saw earlier. That is a distribution over all possible tokens, its prediction for what comes next.
This is how generation works. We pick a token and append it. The diagram gains a new column for that position, using the same model layers. The model then predicts the next token, and the process repeats.
Text becomes tokens, and each token becomes a learned vector. From those vectors, the model predicts what token could come next.
Attention combines information from the current and earlier positions, while feed-forward layers process each position. Together, the layers compute the next-token prediction.
That is the foundation. Everything the agent does, every reply and every tool call, passes through this one predictor. In the next video we follow that one prediction onward, through the generation loop, into chat, and up against the context window.