This video looks at chain-of-thought prompting and expert personas. We'll compare what studies found with the claims people make about these techniques. Then we'll test a prompt change against a benchmark. This is the approach you will use in the lab to find out what helps on your chosen problems.
In the previous video we shaped what the model answers. The next technique is about how it gets there. Let's take a small word problem. A pen costs two dollars. A notebook costs three dollars more than a pen. And we want to know the cost of two notebooks and a pen. The answer shown here is eight, which is wrong. It illustrates what can happen when a model gives an answer without useful intermediate working.
Now we ask it to reason step by step instead. A pen is two dollars, a notebook is five dollars, and two times five plus two comes to twelve dollars. In this example, writing out the intermediate steps leads to the correct answer. Asking for this working is called chain-of-thought prompting. The steps still need checking, but they give the model more room to work through the problem.
Research has shown that this works. Worked examples that include their reasoning steps improve results on reasoning tasks. In fact, further research found that even without providing examples, adding the single line "let's think step by step" improves accuracy on the same model. That is why think step by step became widely used. But the idea did not stay a prompting trick. It was built into the models we use today.
Reasoning models are trained to write out intermediate working before the final answer. This working is made of thinking tokens, and it is a direct extension of chain-of-thought prompting. A thinking or effort setting controls how much of it the model does, so you do not need a stock phrase to switch it on.
Another common suggestion is to give the model a persona, such as you are an expert Python developer. The idea behind it is that telling the model it is an expert will make it answer like one. So the honest question is, does that actually hold up when you measure it?
Some studies found improvements. Kong and colleagues used role-play prompts with ChatGPT and improved scores on reasoning benchmarks. On an algebra word-problem test, accuracy rose from about fifty-four percent to sixty-four percent. Why might a role help? Recall that the model learned to predict text from examples that include both good and bad answers. One possible explanation is that asking for an expert shifts the next-token probabilities toward patterns of careful expert writing. That is a proposed mechanism, not a guarantee of better reasoning. In more modern models , later training phases also shift the language model toward high quality answers, so a persona may have less left to add. That later phase happens after the model has learned to predict text, and it is where the model is tuned to follow instructions.
We can make the instruction more concrete. Act as a senior Python developer, prefer the standard library, and write typed code gives the model specific requirements to follow. The useful details here are the scope and style of the work. We can check whether the code uses the standard library and includes types, regardless of the expert label.
Use a role to say what kind of work you want. The same code gets a different review from each of these roles. Reviewing code for a first-year course means plain explanations, aimed at what a beginner would find confusing. A UX reviewer looks at what the user sees, such as error messages and labels. A senior developer looks at the architecture, and at how easy the code will be to maintain. A bigger title, like world-class genius programmer, does not describe a different job. While it may have worked in the past on earlier models, it is probably not going to help much today.
Many prompts are written once and used once. Others are reused many times, inside a skill, or sent by your software on every request. For a prompt like that, it pays to treat prompting as an engineering activity: make a change and measure its effect. The first thing you need is a benchmark. It is a set of test prompts with agreed answers or scoring rules. Decide what counts as correct before running the comparison, and write it down. In the first case, the agreed answer is twelve, and we decide in advance that twelve dollars counts too. Keep that set fixed for the comparison.
With the benchmark in place, change one thing in the prompt. Then you know what caused any difference in the scores.
Run the whole set several times, with the same model and settings. A single run can be a fluke, because the model samples its answers.
Then compare the scores. Keep the change if it helps across the runs, and revert it if it does not. Look at which cases changed too, because a higher total can hide a new failure.
When the model itself changes, run the same benchmark again with the prompt unchanged. What helped one model may do little for another.
This loop is the skill that lasts. It gives you evidence as the tools change, instead of a phrase someone recommends.
Chain-of-thought prompting helped in the studies we saw, and reasoning models now build it in as thinking tokens. So check what an added request for steps changes in your setup.
A persona can describe a role or style. Its effect on accuracy depends on the model and task, so an expert label still needs testing.
You now have a way to evaluate prompting advice: choose a benchmark, change one thing, and compare repeated runs. In the lab, use that process to find out what helps and where failures remain.
That is the last video for this week. See you in the lab, and happy prompting.