The models we've used through coding agents have been running on a provider's servers. In this video we'll run open-weight models on a laptop, where we can send requests and inspect the responses ourselves. We'll compare a base model with a chat model, examine the chat template, and look at a community model trained with refusal examples filtered out. These examples help explain what post-training changes and why we also need permissions around an agent's actions.
We'll use Ollama to download and run open-weight models. Open weights means the trained parameter values are available to download. Ollama exposes the locally running model through an HTTP endpoint, normally on port eleven thousand four hundred and thirty-four. The downloaded weights are stored on disk. Ollama loads them into memory for inference and accepts requests through its server.
Install Ollama if you want to follow the demonstrations. On Linux and macOS, the command on screen downloads and runs the installer. The website also offers ordinary downloads, and Windows has its own installation instructions. Choose the option for your operating system. Run ollama list to see the models already downloaded. This capture lists three, in the order shown: a community fine-tune of Llama 2 trained without refusal examples, the eight-billion-parameter Llama 3.1 chat model, and a one-billion-parameter Gemma 3 base model, pulled from Hugging Face as a quantized GGUF file. Ollama run followed by a model name downloads it if necessary and opens a terminal conversation.
Ollama's model catalog is available at ollama dot com slash search. You can browse model families and sizes there before choosing one that fits your machine. The outputs in this video are saved runs from the date shown on each slide. Pause at a command and try it yourself if you like. The exact response and speed may differ, which is part of what makes these comparisons useful.
The small models here are practical counterparts to the hosted models we've been using. Speed depends on the model and the hardware, and generation can be slow on a laptop running only on its CPU. With local inference, the prompt is processed on your machine. Downloading weights uses the network, and Ollama also supports cloud models, so make sure you have selected a local model when local processing is the point of the exercise.
Our first example is the pre-trained version of Gemma, a base model without the later instruction-tuning stage. We introduced pre-training earlier in the course as learning to predict continuations of text. We give it the beginning of a sentence, the capital of France is. It continues with Paris and then more prose about the city, including an inconsistent claim. Producing a plausible continuation does not establish that the continuation is true.
With the question, what is the capital of France, this recorded run continues with more questions instead of answering. One even asks why France has three hundred and sixty days a year, which is itself a nonsensical question. A list of questions is one possible continuation of a question in ordinary text. The model is generating a continuation, and this prompt did not reliably establish the question-and-answer behaviour we wanted.
Here the base model continues with more questions. Its pre-training task was to continue text, including any dialogue in that text. Post-training develops the assistant behavior we want: answering the question.
The chat version of Llama three point one answers the same question directly: the capital of France is Paris. That is the behaviour we normally expect from a chat assistant. Instruction tuning is part of the post-training used to develop that behaviour. This comparison uses different model families and sizes, so it illustrates the distinction rather than isolating post-training as the only experimental difference.
The input format matters too. Ollama applies the model's chat template, which marks the roles and turn boundaries before the model continues the sequence. The user's header and question are shown in amber. The end-of-turn marker closes that turn, and an assistant header tells the model which role to continue. The assistant's answer is shown in purple. The model generates the assistant's text and can emit the end-of-turn marker it learned to use. The serving software recognizes that marker and stops generation. An output limit can also stop a reply.
The next comparison asks how to pick the lock on the user's own front door. In this saved run, the official Llama chat model refuses and suggests contacting a locksmith.
The community model called llama two uncensored responds to the same request with general advice. We are comparing the willingness to answer, not evaluating whether its advice works.
The model's origin matters. This release was fine-tuned from the Llama 2 base weights using a conversation dataset whose refusal responses had been filtered out. It was not made by removing a component from Meta's already-trained chat model. Filtering those examples changes what the fine-tuning stage encourages. Access to the base weights makes that alternative training path possible. We can then compare how the resulting models respond.
The diagram summarizes the training paths. Pre-training on a large text collection, including web text, produces a base model that predicts continuations.
One path then trains the model for chat, including answering requests and refusing some of them. Those response patterns are learned behaviour in the weights.
The other path fine-tunes the Llama 2 base model using the filtered conversation data. The alternative fine-tune responds differently to some requests.
Training is one way to make a model respond more safely. It can teach useful patterns for recognizing requests it should decline, and those patterns affect what the model produces. Those learned responses are not a hard limit on actions. They can fail, and people with the weights can change them through further training. Permissions and oversight add a separate layer that controls which operations the harness may actually carry out.
The catalog contains a wide range of sizes. Models with around one or two billion parameters are useful starting points for CPU experiments, although the card includes some larger examples too. Check memory requirements as well as download size.
Some models accept images as well as text. For those, you can ask about an image within the same general request-and-response workflow.
Several major labs publish open-weight model families. Examples include Meta's Llama, Google's Gemma, OpenAI's gpt-oss, and models from Mistral. Their sizes and capabilities vary considerably.
The catalog also includes releases from Chinese labs, such as Alibaba's Qwen and DeepSeek. These are further examples of the range of models available to run or study.
Community developers publish fine-tunes, distilled models, and merged models as well. Ollama can also download supported GGUF models straight from Hugging Face by their repository address. That is how the Gemma base model in our first capture was fetched, which is why its name on the list began with hf dot co. The file format alone does not guarantee that every architecture is supported.
For a supported model that fits your hardware, ollama run followed by its name is enough to get started. A model name refers to a particular release and configuration, so check which version you are downloading when comparing results.
Have fun downloading a few models and playing around with them. Try the same prompt on a base model and on a chat model, and see how differently they answer. Just remember to delete them when you are done, with ollama rm followed by the model name, if you want the space back on your machine.