You ran your own experiments in Lab 2. Here is what the whole class found, put together from every report. All findings are anonymous.
Where the models made mistakes
What you tested
Almost half of you chose one of the problems the lab suggested: multiplication, letter counting, facts or theory of mind. The rest went everywhere, from chess to Hebrew riddles. We grouped the reports by the kind of problem they tested.
Reports by kind of problem
Number of reports. Our grouping, from each report's description of its task.
Show as a table
From the long tail
A few of the less common experiments, and what they found.
Facts
“What weekday was 30 February 1712 in Sweden?”
That date really existed: Sweden briefly ran its own calendar. Haiku answered that February never has 30 days, in all 10 runs.
Facts
“Is there a seahorse emoji?”
There is not, and the models never invented one (0 of 44 runs). But they missed a real emoji newer than their training, the Emoji 18.0 lighthouse, in 28 of 28 answers.
Writing under constraints
“Write the word ‘ocean’ exactly 200 times.”
Haiku stopped early (149 to 175 words). Opus overshot (209 to 218). When the words were written in groups of ten, the counts became exact.
Writing under constraints
“Write five facts about elephants without the letter e.”
Some answers passed by cheating: the model used a Cyrillic “е”, which looks exactly like “e” but is a different character.
Trick questions
“I want to wash my car. The car wash is 100 m away. Should I walk or drive?”
The answer is drive: the car itself has to get to the car wash. But a short distance usually means “walk”, and models often answered that. Eleven reports tested this trap. In one, adding “It's a nice day and I want exercise” sent Haiku and Sonnet to 0 of 5, even with step-by-step working. Opus stayed perfect.
Facts
“Which of these share a land border with France: Spain, Brazil, Netherlands, Suriname?”
All four do, through French Guiana and Saint Martin. Told to answer with one letter, the model said only “A” in all 20 runs. Without that rule, the newer models named all four in 105 of 120 runs.
Multiplication
“What is one hundred twenty-two times five hundred twenty-eight?”
Written in digits: 5 of 5 right. Written in words: 0 of 5, always 64,456 (the answer is 64,416). With step-by-step working, the model first rewrote the words as digits, in 90 of 90 runs.
Theory of mind
Where will Sally look for her marble?
One report generated stories with control versions, where the usual answer is wrong (for example, Sally secretly watches the marble being moved). Scores on the classic stories were high. On the controls they collapsed: one model scored 0 of 15. A high score can come from a shortcut.
Logic
“Score this basketball play-by-play.”
Four traps were planted in the text, such as two players with the same shirt number. Haiku fell for them in all 10 runs. Opus avoided every trap but disagreed with the answer key on one disputed foul.
Also tested:
- Find the mate in two
- Predict what this Python code prints
- DNA reverse complement
- Factor a 19-digit number, no code
- Add 1,000 hours to a clock time
- Hebrew wordplay riddles
- The coin rotation paradox
- Cups and balls through 256 swaps
- Card position after riffle shuffles
- Seating puzzles with 10 people
- The pirate game
- Populations of Israeli towns
- Read a noisy ASCII-art captcha
- Multiply 5×5 matrices
- Ten words that fit on a keyboard's top row
The class curve for multiplication
Correct answers by number size
Percent correct, by digits in each of the two numbers (4 means 4 digits × 4 digits). Hover for the counts.
Show as a table
Did step-by-step working help?
The effect of step-by-step working, by kind of problem
Number of reports, as each report concluded. “Mixed” means it helped one model or size and not another. Reports that did not test it are left out.
Show as a table
When writing it out made things worse
Seven reports found that working made a model worse. The interesting part is why. In each case, the working changed what the model was actually doing.
Multiplying in base 5
Haiku: 40/75 → 27/75 with working
Told to give only the answer, Haiku ignored the rule and quietly converted the numbers to ordinary decimal in its reply. That worked. Told to show its working, it did long multiplication in base 5 instead, and made carry and alignment slips. It even wrote the digits 5 and 7, which do not exist in base 5.
Cups and balls
Haiku at 40 swaps: 9/10 → 0/10 with working
The question asked about one ball. The step-by-step prompt said to write down where each ball is after every swap. So Haiku tracked three balls instead of one, and made more slips. (Most of the “answer only” replies in this report wrote some working anyway.)
Counting a Hebrew letter
Sonnet: 5/5 → 1/5 with working
The task was to count כ in a word, with the final form ך counting as the same letter. Spelling the word out one letter per line put ך on its own line, and the model then treated it as a different letter.
The car wash
Haiku: 11/30 → 6/30 with working
The question: “I want to wash my car. The car wash is 100 m away. Should I walk or drive?” The right answer is drive, because the car must get there. Haiku's working was a short argument for the wrong answer: 100 m is short, so walk. Working gave the wrong idea room to justify itself. In a version about filling the car at a gas station, Haiku wrote “drive only to fill up gas” and then answered “walk”. The same step-by-step prompt helped Sonnet (20/30 → 25/30).
Four things many of you found on your own
Some findings came up again and again, in reports that tested different problems. The counts are reports that described the effect themselves, so they are a minimum.
The model ignored “answer only”
What you saw
Many prompts said “reply with only the final number”. The model often wrote out its working anyway. Opus did this most: in one report it ignored the rule in all 30 answer-only runs, in another in 25 of 30. Haiku did it more as problems got harder: in one report it wrote working in 0 of 16 replies on the easiest level, but 12 of 16 on the hardest.
Why it matters
Your “no working” test was then a test with working, and the scores looked too good. One report checked this. Opus got 15 of 15 at 6 and 8 digits “without working”, because it wrote working. With a strict prompt (“Do not write any working… your whole reply must be a single line”), it fell to 5 of 15 at 6 digits and 0 of 15 at 8. In another report, every bare-number answer to a 4-digit or larger product was wrong: 0 of 91.
Why it happens
The text a model writes is its scratch paper. Each new word is computed from everything written so far, so writing steps out gives it room to compute. A model trained to get answers right reaches for that room, even against your instruction.
For your own work: an instruction is a request, not a guarantee. Read what the model actually wrote, not only the final number. Count how often it followed the format, and score the two kinds of reply separately.
Wrong answers that look right
What you saw
Wrong products were rarely wildly wrong. The first few digits were right, the last few were right, and the middle was invented. One report checked all 27 wrong answers of the right length: every one had the first two and last two digits right. Another found wrong answers within 0.1% of the truth, always with the right last digit. Here are three wrong answers from one report:
Why it happens
The last digit of a product depends only on the last digits of the two numbers, so it is easy. The first digits follow from the rough size of the answer, which is easy to estimate. The middle digits depend on every carry from every partial product. That is the part that needs real step-by-step work, and it is where the model guesses. One report caught this directly: Haiku wrote each partial product correctly, then added five or six of them in one go without writing the sum (in 46 of 90 answers).
Why it matters
The usual sanity checks pass these answers: the size is right and the last digit is right. Two reports even saw a model's own “casting out nines” check approve a wrong answer.
For your own work: a plausible answer is not a checked answer. When exactness matters, check exactly: run the code, or compute it another way.
A tool call that never ran
What you saw
Tools were turned off. Even so, some replies contained what looked like a tool call, followed by its “output”, and then an answer based on that output. None of it ran.
- Base-5 arithmetic: in 4 of 5 hard runs, Opus wrote a Python call, invented its output and reported it, giving four different wrong answers. The one run that worked it out by hand was right.
- 9- and 12-digit products: Sonnet 4.6 wrote a fake Python tool call, invented the “output” and reported that number. Both numbers were wrong.
- Cups and balls at 128 swaps or more: Haiku wrote a script and announced “the simulation is complete” in 31 of 40 runs. Only 7 of those 31 were right, which is about what guessing gives with three cups.
A close relative: one model wrote a guessed number, then “Let me verify”, and put ✓ marks on its own working. The errors were never caught.
Why it happens
In an agent, the model asks for a tool by writing a tool call. The agent program, such as Claude Code, runs it and pastes the real result back into the conversation. Here nothing ran it. The model just kept writing, and after a tool call, the most likely next text is something that looks like the tool's output. So it wrote that too.
“This is the most dangerous kind of agent mistake: the answer looks verified by a tool when no tool ran.”
For your own work: text that looks like output is not output. When it matters, check the agent's log for a real tool result.
Recalled, not solved
What you saw
When a question looked like a famous one, the model often gave the famous answer, whether or not it fit.
- An olympiad problem. Two reports, separately, tried Problem 6 of the 2025 International Mathematical Olympiad on grids of size 4, 16, 100 and 2025. The correct answers are 5, 21, 117 and 2112. The models failed every small grid, often answering 2N − 2 (6, 30, 198). But Opus answered 2112 for the 2025 grid in all 4 runs. The 2025 grid is the published competition problem.
- A puzzle with its numbers missing. Two reports sent the bat-and-ball puzzle (“a bat and a ball cost $1.10…”) through a shell script, and the shell swallowed the dollar amounts. The question no longer had an answer. The models gave the famous one anyway: 25 of 30 runs in one report, all 30 in the other.
- A riddle with one detail changed. In the classic riddle, a driver sees a man in black on a dark road, and the answer is “it was daytime”. Two reports set the riddle at 2 AM on a moonless night. Haiku still answered “it was daytime”.
- The pirate game. Models solved the textbook five-pirate version, but failed versions with other numbers when answering directly, such as one with too few coins for the first pirate to survive.
The flip side showed up too: famous riddles were answered right in 90 of 90 runs, in the same reports where the less famous car-wash question (walk or drive to wash your car) failed.
Why it happens
Famous problems and their answers appear many times in the text a model learns from. When your question looks like one of them, the famous answer is the most likely continuation. Remembering is easier than solving.
For your own work: success on a famous problem tells you little. Test with new or altered versions. That is why the best reports generated fresh problems for every run.