This video continues our explanation of how large language models work. We will see how post-training prepares a model to act as an assistant, then look at model sizes, training data, and the computation needed for training. We will also look at how models answer requests over the network, how large context windows have become, and how developments since AlexNet led to the language models we use today.
In the previous video, we trained a neural network on examples by measuring a loss and updating the weights to reduce it. For a language model, the examples are sequences of tokens. The training task we met in Week 2 is to predict the next token from the preceding text. Much of that text comes from the public internet. This initial training phase is called pretraining. The resulting model generates continuations of its input, one token at a time. Pretraining alone does not reliably make it behave as a helpful assistant. Given a question, it might answer, or it might continue with more questions written in the same style.
Further training prepares the model for use as an assistant. Fine-tuning is part of this broader phase, which is called post-training. One approach uses a smaller, carefully selected collection of assistant conversations. Another uses feedback, such as human judgments about which of two answers is better, to encourage preferred responses. These methods continue adjusting the model's weights, using different data and objectives from pretraining.
Post-training strongly shapes the behavior we encounter when we talk to the model. It encourages helpful and polite replies, instruction following, refusals of harmful requests, and a consistent tone across a conversation. Pretraining already affects behavior, but post-training gives these assistant behaviors more explicit training. It does not guarantee that the model will follow them every time.
Pretraining develops broad knowledge and capabilities. Post-training builds on them to make the model more useful as an assistant. This is a useful distinction, although both phases can affect knowledge and behavior. When the model follows a request, responds cautiously, or maintains a consistent style, we are seeing behavior that post-training was designed to encourage.
Let's put some numbers on the scale of these models. We will use Meta's Llama family because Meta publishes parameter counts, architecture details, and training-data totals. Those figures give us specific examples to compare. Llama two, released in twenty twenty-three, had a largest model with seventy billion parameters. In twenty twenty-four, the Llama three family reached four hundred and five billion. Parameters include the learned weights we discussed in the previous video. For comparison, the human brain has on the order of a hundred trillion connections, called synapses. That is more than the parameter counts here, although a biological synapse and an artificial weight are different things. The comparison gives scale, not a measure of intelligence. Across Llama two, three, and four, the vocabulary, the set of tokens the model reads and writes, grew from about thirty-two thousand tokens to a hundred and twenty-eight thousand, then about two hundred and two thousand.
The training data grew even faster. It went from two trillion tokens for Llama two to about fifteen trillion for Llama three. Llama four Scout used around forty trillion, while Maverick used around twenty-two trillion. To get a feel for these numbers, a token is roughly three quarters of an English word. So a book of a hundred thousand words is about a hundred and thirty thousand tokens. The Library of Congress holds about twenty-six million cataloged books, roughly three trillion tokens. In twenty ten, Google estimated that about a hundred and thirty million books had ever been published. That is around seventeen trillion tokens, close to the amount Llama three was trained on. Epoch AI estimates that all the public text on the internet comes to around three hundred trillion tokens. Llama three's training data is already about a twentieth of that.
Llama four introduced a mixture of experts to the Llama family. The two models released in April twenty twenty-five were Scout and Maverick. A mixture of experts has expert subnetworks, and routes each token through only some of them. Maverick has about four hundred billion parameters in total, with about seventeen billion active for each token. This reduces the computation per token compared with using all of those parameters, while the full model still needs storage. It also allows a larger total parameter count. At the April release, Meta described Behemoth, with nearly two trillion parameters, as still in training.
Where does the training text come from? The public internet is an important source, alongside licensed material and other sources available to the developer. Common sources for language-model datasets include web crawls such as Common Crawl, public code repositories such as GitHub, Wikipedia, books, scientific papers, and question and answer sites. For Llama three, Meta reported a mixture of roughly fifty percent general knowledge, twenty-five percent math and reasoning, seventeen percent code, and eight percent multilingual tokens.
Llama four Scout used around forty trillion training tokens, more than twice Llama three's total. Meta described several changes to the data. The training mixture included images and video as well as text. It also used about ten times as many multilingual tokens as Llama three, covering two hundred languages. The model card also lists data from Meta's products: publicly shared Instagram and Facebook posts, and people's interactions with Meta A-I. It gives an August twenty twenty-four cutoff for the pretraining data.
Meta published source categories and training totals for Llama four, but not a numerical breakdown comparable to Llama three's mixture. We therefore cannot assign the roughly twenty-five trillion additional tokens in Scout's training to particular sources. Knowing that a source was included does not tell us what proportion it contributed.
To see how this data becomes training examples, let's return to the training loop from the previous video and apply it to text. We take a chunk from the training corpus. The last token here is mat. It provides the correct next-token answer, and is hidden from the model for this prediction.
The model receives the preceding tokens, the cat sat on the, and produces a probability for each possible next token. It assigns mat a probability of forty percent.
We compare that prediction with the token in the original text, mat. The loss penalizes the model for assigning the observed token too little probability. Language models commonly use the negative logarithm of that probability, rather than the squared gaps in our digit example. Increasing the probability of mat reduces this example's loss.
We use the loss to compute updates to the weights. As in the previous video, the aim is to reduce the loss so that predictions improve over the training examples. Backpropagation computes the derivatives needed for those updates throughout the network. An optimizer uses them to change the weights. We will leave the detailed mathematics to a machine learning course.
We repeat this process across billions of next-token examples from the corpus. In practice, training processes many examples together in a batch.
The text provides something that we had to supply separately for our digit images: a correct training target. The next token is already in the original text. We can construct each prediction and target pair from that text, without asking someone to label each example.
This is called self-supervised learning: the training targets come from the data itself. Avoiding a separate human label for every example makes it possible to use very large text collections. Gathering, licensing, filtering, and processing the data still costs money, and suitable fresh text is finite. The large supply of examples with automatically available targets is what makes next-token training possible at this scale.
Coding is one of the tasks these models do best, and the way they are trained helps explain why. The first reason is data. Public code repositories hold an enormous amount of code, and about one in six of Llama three's training tokens was code.
The second reason is that a program can check the answer. Run a proposed solution against its tests, and you learn whether it works without asking a person. So post-training can reward correct solutions automatically, across a huge number of programming problems. This approach is called reinforcement learning with verifiable rewards. Like the next-token targets we just saw, these rewards need no human label for each example, so this kind of training can grow very large.
So coding offers plenty of examples to learn from, and answers that a program can check. The same checkability helps an agent while it works, because it can run the code, read the result and try again. The same reasoning reaches beyond code. Many math answers can also be checked automatically, and so can proofs of theorems, using a program called a proof checker. That is part of why models are strong at math. Expect them to keep improving there, and at any other task where a program can check the answer.
Working through all those training examples requires a large amount of computation. Meta reported about thirty-one million H-one-hundred GPU-hours for Llama three's four-hundred-and-five-billion-parameter model, using up to about sixteen thousand GPUs. At that GPU count, the total is equivalent to roughly eighty days, or two to three months. To get a feel for that number, we will spread the same computation over less hardware. This is an order-of-magnitude thought experiment, not a benchmark. We hold the useful work per GPU-hour fixed.
Dividing roughly thirty-one million GPU-hours by the hours in a year gives about three thousand five hundred GPU-years. One GPU would need that long to do the work, if it could hold the model in memory. Imagine starting one chip three and a half thousand years ago and leaving it running until today. That is the amount of computation the cluster spreads across thousands of chips.
A single CPU core performs much less of this parallel matrix arithmetic per second. Suppose the core were seventeen thousand times slower than one training GPU. That is the speed ratio we are assuming for this comparison. Three thousand five hundred GPU-years then become about sixty million CPU-core-years. Imagine one core running almost all the way from the extinction of the dinosaurs to today.
Parallel computation brings that work down to months. The chips share the calculations and exchange intermediate results, so keeping them working together efficiently is part of the engineering challenge. For a rough rental comparison, thirty-one million GPU-hours at about three dollars per GPU-hour comes to around a hundred million dollars. At about seven dollars per GPU-hour, it is roughly two hundred and twenty million dollars. Those assumed rental rates give us a sense of the cost, rather than Meta's actual bill. These estimates cover a training run's GPU time. Developing a model also involves experiments and other costs, so the full development cost is larger. Mixture of experts is one way to reduce computation per token. Meta reported about five million GPU-hours for Llama four Scout, and about two point three eight million for Maverick, compared with about thirty-one million for Llama three's largest model. Architecture, data, hardware, and the rest of the training setup differ too.
Training produces the weights we use for inference. Answering a request with the trained model also requires computation and memory, even though it does not update the weights. In the setup shown here, your application sends a prompt over the network to a cloud A-P-I. The provider runs the model on GPUs or other accelerators and streams generated tokens back. A frontier model of this size needs more hardware than an ordinary laptop provides. Smaller models can run locally, but large hosted models are normally accessed through a service.
Much of the computation in a neural network consists of matrix multiplications and applying operations to many neuron outputs. GPUs can perform many of these operations in parallel. Large models also need substantial memory to hold their weights and the intermediate state used while generating a response.
Serving a model efficiently involves several kinds of engineering. KV-caching stores intermediate results for previous tokens so they do not have to be recomputed. Batching combines work from several prompts, custom GPU kernels speed up operations, and quantization represents numbers with fewer bits to reduce memory use and computation. We will not develop those implementation details here. They are active areas of research and engineering. For the large hosted models we use in this course, the practical result is that our programs call a cloud A-P-I instead of loading the model on a laptop. A request to such a model is therefore a network request to a service running the model on accelerator hardware.
The context window is another scale limit you will encounter while working with agents. In Week 2, we defined it as the maximum number of tokens available to the model in one request. Llama4-Scout supports a context window of ten million tokens. Claude Opus five point five and Fable five point one support one million tokens in the documented offerings. A long session can still reach those limits as files and tool results accumulate. That is why the harness compacts conversations, as we saw in the video on context.
How did neural networks get from recognizing images to the coding agents we use today? A useful place to start that story is AlexNet, a neural network that won an image recognition competition in twenty twelve. Its top-five error rate in the ImageNet competition was fifteen point three percent, compared with twenty-six point two percent for the next-best entry. This large improvement helped establish deep neural networks as a practical approach to image recognition.
The authors also connect this early result to the labs that later developed widely used A-I systems. Geoffrey Hinton and Alex Krizhevsky joined Google. Ilya Sutskever went on to co-found OpenAI. Their later work is one connection between the progress in image recognition and the development of modern language models.
Over the following years, deep neural networks became a standard approach across many computer-vision tasks. In twenty seventeen, the Transformer introduced an architecture based on attention that became central to language modelling. We will not derive the Transformer here. It is the architecture used by many modern language models, and attention is one of its central mechanisms.
Large language models and coding agents developed from these advances in neural networks, training methods, and computing hardware.
A repeated pattern in this history is that more data and computation made larger networks useful on more demanding tasks. Since twenty twelve, scaling neural networks has helped improve image recognition, language modelling, and the assistants we now use. Architecture and training methods improved alongside that increase in scale.
This chart from Epoch A-I shows the growth in training computation. Each point represents a notable model. The horizontal axis is time, and the vertical axis gives training computation in floating-point operations. The vertical scale is logarithmic: each marked order of magnitude represents ten times as much computation. The fitted trend summarizes rapid growth across the models studied. Epoch's twenty twenty-four analysis estimated about four to five times more training computation per year from twenty ten to twenty twenty-four.
Epoch's earlier analysis described an increase of roughly ten orders of magnitude since twenty ten. That means about ten billion times as much training computation over the period studied. This is much faster than the growth in computation available from a single chip. Labs also devoted more chips, time, and money to training, which helps explain the scale of the models we have discussed.
The biggest models keep getting more expensive to train. But the price of a given level of quality keeps falling. Here is one way to see that. Pick a fixed test score, and ask what the cheapest model that reaches it costs to use. This chart does that for MMLU, a common test of general knowledge. Each step down on the price axis is ten times cheaper. The purple line follows the cheapest model that scores above forty-two. In November twenty twenty-one, that was G-P-T three, at sixty dollars per million tokens. By late twenty twenty-four, a small Llama model with three billion parameters reached the same score for six cents. That is a thousand times cheaper in about three years.
Several changes help explain why a fixed level of performance became less expensive. Better training methods let smaller models reach performance levels that previously required larger ones. Caching, batching, and quantization reduce serving costs. Competition among providers can also lower the prices users pay. This helps make coding agents practical, because an agent may call a model hundreds of times for a task.
Let's recap. The pretraining objective we followed was to predict the next token from the preceding text.
For Llama three's largest model, that meant hundreds of billions of parameters, about fifteen trillion training tokens, and millions of GPU-hours. Those numbers show how a simple training objective can involve a very large computation. The text supplies its own next-token targets, so the training set does not need a separate human label for every prediction. This makes it possible to train on far more examples.
Training at that scale develops broad capabilities. Post-training then shapes how the model uses them as an assistant, and inference makes those capabilities available to our programs. These are the stages behind the language models we work with today. Their scale and complexity are truly staggering and continue to grow. See you in the next videos!