AICODEAgentic Programming · 67723 · 2026-7

Week 2 · lab handout

Find Where the Model Makes Mistakes

20–22/10 · 90 minutes, in two halves · in pairs · your section's day and room

Goal: Find where advanced models make mistakes, and learn how prompting, training, and tools can reduce those mistakes.

Work in pairs, one laptop each. Your partner must be someone you did not work with in Lab 1. This is strictly enforced. Choose what to test and write your predictions together. Each of you runs your own experiment and writes your own report. Discuss choices and help each other when you need help.

Phase 1 — Explore and choose a problem (first 15 minutes)

Choose one kind of problem with your partner. Different pairs will investigate different problems, and we’ll compare findings at the end.

1. Try the model. Start in an empty directory. Open the box for the agent you installed, set PROMPT to a question you can check, run the command, and read the answer.

With Claude Code 
PROMPT="your question"
echo "$PROMPT" | MAX_THINKING_TOKENS=0 claude -p --model claude-haiku-4-5 --tools "" \
    --strict-mcp-config --system-prompt "You are a helpful assistant."

Models: use the newest model in each line that can turn separate thinking off:

  • Haiku: claude-haiku-4-5
  • Sonnet: claude-sonnet-5
  • Opus: claude-opus-5

Newer models keep thinking even with MAX_THINKING_TOKENS=0. That includes every Fable model and the short names haiku, sonnet, opus and fable, which select the newest versions.

What the flags do
  • | sends the prompt through standard input (stdin). -p prints the answer and exits.
  • --model claude-haiku-4-5 selects the exact model version.
  • --tools "" disables built-in tools. --strict-mcp-config ignores your configured MCP servers, so their tools are not sent either.
  • MAX_THINKING_TOKENS=0 disables separate thinking on the models listed above. The model can still write steps in its response. Put this variable before claude, not before echo.
  • --system-prompt replaces the coding agent’s default instructions with the text shown. Claude Code still adds a few notes to your message, such as today’s date and your CLAUDE.md.
With Codex

Save the system prompt to a file once:

printf 'You are a helpful assistant.\n' > ~/sys.md

Then run:

PROMPT="your question"
echo "$PROMPT" | codex exec --ephemeral --skip-git-repo-check \
    -s read-only -m gpt-5.6-sol -c model_reasoning_effort="none" \
    -c model_instructions_file=$HOME/sys.md -c web_search="disabled" \
    --disable shell_tool --disable unified_exec --disable view_image \
    --disable multi_agent --disable apps --disable plugins \
    --disable skill_search --disable tool_suggest --disable sleep_tool \
    --disable browser_use --disable computer_use --disable image_generation

Models: Start with gpt-5.6-sol. For model comparisons, you can also use gpt-5.6-terra or gpt-5.6-luna. All three support none; keep that setting for this experiment and record the exact model name used.

Add -o answer.txt to write only the final answer to a file. A test program can read it from there. Add --json to record tool activity too. Discard and repeat any run that uses a tool.

What the flags do
  • codex exec prints the answer and exits. The prompt arrives through stdin.
  • -m gpt-5.6-sol selects the model.
  • -c model_reasoning_effort="none" disables separate thinking. The model can still write steps in its response.
  • -c model_instructions_file=... replaces Codex’s default instructions with the file’s text. The path is relative to the directory you run from, so use the full path.
  • -c web_search="disabled" and the --disable flags turn off most built-in tools. Two tools remain available (apply_patch, request_user_input). -s read-only restricts file changes, but does not prove that no tool was used; check the tool activity.
  • --ephemeral does not save the session. --skip-git-repo-check lets Codex run in a directory that is not a git repository.

An empty directory avoids project instructions, but user-level instructions and saved memories can still load: Claude reads CLAUDE.md files up the directory tree and in ~/.claude/, Codex reads AGENTS.md files and ~/.codex/config.toml. Check what your setup loads and keep it the same across conditions.

2. Choose a problem. Try a few examples whose answers you can check. Change the numbers, words, or details to make your own examples, then choose one problem type to study.

Ideas for problems
  • Multiplication. Ask for the product of two large numbers, with only the answer in the response.
  • Letter counting. Ask how often a letter appears in a word or a long string.
  • Facts and hallucinations. Ask a factual question and check the answer against a reliable source. Does the model give a wrong answer confidently?
  • Theory of mind. Ask what someone knows or believes. For example: Joe puts his keys in a box and leaves. Fred moves them to a closet. Where will Joe look when he returns?
  • Your own idea. Any other problem whose answer you can check.

3. Write three predictions together. For your chosen problem type:

  1. How difficult must the problem be before the model starts giving wrong answers?
  2. Will asking for step-by-step working (chain-of-thought, CoT) help?
  3. Will a harder problem cause the model to give wrong answers even with CoT? How hard?

Tell the lab instructor which problem you chose. Stay with it for the rest of the lab.

Phase 2 — Run your experiment (until 30 minutes before the end)

Design an experiment and run it. Stay with your chosen problem type and look for an interesting result to share. Use your predictions to decide what to investigate. You could compare:

  • Models: the ones listed in your agent’s box above.
  • Problems: different examples at the same difficulty, or increasingly difficult examples.
  • Prompts: with and without chain-of-thought working.

Choose comparisons that interest you; you do not need to try every combination. Change one thing at a time and keep tools and separate thinking disabled.

Use your agent to build the test program. Decide how to check answers before running it, then inspect a few responses to make sure it scores the final answers correctly.

Repeat each input at least five times, with the same settings and a new conversation each time. Answers may vary. Record how often the model answers correctly, including cases where it always succeeds or always fails.

This is uninterrupted work time. Help your partner and ask the lab instructor when you need help.

Phase 3 — Generate and compare reports (30 to 15 minutes before the end)

Stop adding tests and generate a report from the results you have.

Each of you writes your own report: half a page maximum, with at least one table or chart. Start with one short TL;DR sentence stating the main result: where the model made mistakes, whether CoT helped, and what happened on harder examples, if you tested them. Have the agent write the report, then check that its numbers and conclusions match your results.

Explain the experiment so that someone who did not attend the lab can understand it. Include the prompts, the exact model version (for example claude-opus-5 or gpt-5.6-sol, not opus), the settings, and the number of runs for each result.

Compare your report with your partner’s. Did you reach the same conclusion? Show your reports to the lab instructor.

Phase 4 — Discussion (last 15 minutes)

Each pair shares one TL;DR sentence with its main finding. We’ll hear from every pair to see what happened across the different problems tested in the room.

If there is time afterward, we’ll discuss the results and what surprised us.

How you’re graded

The grade is determined by:

  1. Report checked by the lab instructor (3 points).
  2. Upload your report as a PDF to Moodle before leaving (1 point): Lab 2 report.

Late for the lab (15 minutes or more): -1 point Unprepared for the lab (for example, no working setup): -1 point or more