The Illusion of Thinking

You told me you are an engineer with no background in AI, machine learning, statistics, neural networks, or reading research papers. This course assumes exactly that and nothing more. Every term gets defined the first time it appears. Every graph gets read for you line by line, as if you had never seen a scientific chart before.

The goal is not that you recognise the paper's claims. The goal is that you can sit across from a colleague, draw on a whiteboard, and teach them the whole thing without notes.

The seven parts

Conventions used on this page

• Dotted terms show a definition on hover.
• Coloured boxes are signposts: Think like this builds intuition, Remember this is memorisable, Common misunderstanding is a trap to avoid.
• Every chapter heading is clickable — click it to collapse that chapter.
• All 13 figures are recreated in live SVG, not screenshotted. Hover any data point to read its value.

Before we can read this paper, you need to know what kind of object a paper is. If you've only ever read documentation and blog posts, a paper looks like a wall of dense text. It isn't. It's a highly structured argument with the same skeleton every time.

The universal skeleton

How to read one efficiently (the three-pass method)

1. Pass 1 (5 min): Title → Abstract → figure captions → Conclusion. You now know the claim.
2. Pass 2 (30 min): Introduction → Method → Results, looking at each figure as it's referenced. You now know the evidence.
3. Pass 3 (hours): Every detail, appendix included, asking "would this convince a hostile expert?" You now have an opinion.

This course is essentially a very slow, very thorough Pass 3, with all the missing background filled in.

Vocabulary you'll meet in any paper

Everything in this paper is about machines called Large Language Models. Let's build that idea from absolutely nothing.

Step 1: The one-sentence version

Yes, really. The chatbot that writes your emails is, mechanically, an extremely sophisticated autocomplete. The interesting scientific question — and the question this entire paper is about — is what capabilities fall out of doing autocomplete well enough, and where those capabilities stop.

Step 2: Tokens — the atoms of the system

Models don't see letters or words. They see tokens. A token is a chunk of text, usually a common word or a word-fragment.

Step 3: How it predicts — probability over the next token

Given the text so far, the model outputs a probability for every possible next token. Probability just means "how likely, on a scale from 0 to 1" — 0 means impossible, 1 means certain, and all the probabilities across all possible next tokens add to exactly 1.

Step 4: Where the numbers come from — training

Inside the model are billions of numbers called parameters or weights. They start random — the model outputs gibberish. Then:

1. Show the model a piece of real text with the last token hidden.
2. Ask it to predict that token.
3. Measure how wrong it was. This measure is called the loss.
4. Nudge every parameter slightly in the direction that would have reduced the loss. (The algorithm for computing those nudges is called gradient descent — you don't need the maths, just the idea: roll downhill on an error landscape.)
5. Repeat, trillions of times, over most of the public internet.

This stage is pre-training, and it is the expensive one — months of computation on tens of thousands of specialised chips. The paper alludes to this: "LLMs undergo multiple costly training phases using vast amounts of training data."

Step 5: What a "neural network" is (you need less than you think)

You don't need the maths, but you should have the picture:

• A neural network is a stack of layers. Each layer takes a list of numbers, multiplies them by its parameters, adds them up, applies a simple squashing function, and passes the result to the next layer.
• Text goes in as numbers (each token maps to a list of numbers called an embedding). Numbers come out as probabilities.
• The specific architecture used by all modern LLMs is the Transformer. Its key trick is attention: when processing any token, the model can look back at every earlier token and decide how much each one matters right now.

The paper repeatedly says things like "under equivalent inference compute" and "well below generation length limits". These phrases carry a lot of weight in the argument. Let's make them precise.

Training vs inference

Inference compute — "how much thinking did it do?"

Inference compute is the amount of computation spent producing one answer. Because generating each token costs roughly the same, counting tokens is a proxy for counting effort. This is the single most important measurement convention in the paper:

Two limits you must not confuse

Around 2022, researchers found something odd and important. Ask a model a multi-step question and it often gets it wrong. Add four words — "Let's think step by step" — and accuracy jumps dramatically. This is Chain of Thought, or CoT, and it's reference [15] in this paper.

Why on earth does that work?

Recall the mechanics: the model generates one token, appends it to its input, and predicts again. Now think about what that means.

There is a second, more technical reason. Without CoT, the model must produce the answer in a fixed number of computational steps — however many layers it has. That's a hard ceiling on how much computation any single answer can involve. With CoT, generating 500 tokens means 500 passes through the whole network. CoT converts a fixed compute budget into a variable one. Harder problem → write more → compute more.

The crucial caveat that sets up this whole paper

If writing out steps helps, the obvious next move is: stop asking politely and bake it into the model. That's what happened in late 2024 and 2025, producing a new class of system the paper calls Large Reasoning Models (LRMs).

How an LRM differs from a plain LLM

Reinforcement learning, in one plain paragraph

Reinforcement learning (RL) is training by trial and reward instead of by imitation. In pre-training the model copies text. In RL, the model attempts something, receives a score, and the behaviours that scored well are made more likely next time. For reasoning models the score is usually simple and mechanical: did you get the maths problem right, yes or no? This is called RL with verifiable rewards.

Crucially, nobody hand-writes the reasoning style. You just reward correct final answers, and long deliberate reasoning emerges because it happens to produce more correct answers. DeepSeek-R1 [3] was the landmark open demonstration of this, matching OpenAI's o1 [2].

What "thinking tokens" look like in practice

Figure 1 of the paper shows a real response structure. Here is that structure, annotated:

<think>                          ← private reasoning section begins
Move disk 1 from peg 0 to peg 2 ...
moves = [
  [1, 0, 2],                     ← candidate solution attempt #1
  [2, 0, 1],
  [1, 2, 1],
  [3, 0, 2],
  [1, 1, 0],
  [2, 1, 2],
  [1, 0, 2],
]
Let me double-check this...      ← self-reflection; may produce attempt #2, #3, …
</think>                         ← thinking section ends. Everything above = "thinking tokens"

<answer> the final answer is moves=... </answer>   ← what a user actually sees

Every paper exists because of a gap. Let's find this one's.

The situation in early 2025

Reasoning models had arrived and were scoring spectacularly on hard benchmarks — competition mathematics, coding challenges, science exams. The paper's own words: "These models are new artifacts... and have demonstrated promising results across various reasoning benchmarks. Their emergence suggests a potential paradigm shift... with some researchers proposing them as significant steps toward more general artificial intelligence capabilities."

Then the paper turns: "Despite these claims and performance advancements, the fundamental benefits and limitations of LRMs remain insufficiently understood."

The four specific questions the authors raise

Straight from the Introduction — memorise these, they organise everything that follows:

Problem A: Data contamination

The smoking gun in this paper

The authors point at a specific, elegant piece of evidence. AIME is a prestigious American high-school maths competition, run annually. AIME24 is the 2024 edition; AIME25 is the 2025 edition — released after most current models finished training.

• Models perform worse on AIME25 than on AIME24.
• But humans performed better on AIME25 than AIME24 [42][43] — suggesting AIME25 was, if anything, the easier exam.

So we have: humans find 2025 easier, machines find it harder. The paper's inference: "potentially suggesting data contamination during the training of frontier LRMs." The older exam had been on the internet longer, so more of it leaked into training.

Problem B: Final-answer-only evaluation

The second flaw is subtler and, for this paper, more important. Standard benchmarks check one bit of information: was the final answer right?

Problem C: You can't turn a difficulty dial on a maths test

This is the deepest problem, and the one that dictates the paper's entire design. Suppose you want to know how performance changes as problems get harder. You need problems whose difficulty you can increase smoothly and precisely, while changing nothing else.

Maths benchmarks can't do this. Is AIME Problem 7 harder than Problem 3? Probably. By how much? Unknown. Is it harder in the same way? Almost certainly not — one might be a geometry insight puzzle, the other a long computation. Difficulty in maths benchmarks is unlabelled, unordered, and multidimensional. In the paper's words, they "do not allow for controlled experimental conditions across different settings and complexities."

Before abandoning maths benchmarks, the authors run one experiment on them. To understand it you need one metric: pass@k.

What pass@k means

The fairness problem this solves

Reading Figure 2 as if you'd never read a graph

Now apply it, panel by panel

A hypothesis is a testable prediction — a statement that could turn out false. Papers don't always state one in a box, so part of reading skill is reconstructing the hypothesis from what the authors chose to measure. Let's do that.

The implicit hypothesis, stated plainly

Notice this is a falsifiable hypothesis. Each of (a), (b), (c) could have come out the other way. That's what makes the experiment worth running. Here's the scorecard we'll fill in over the rest of the course:

The five contributions, decoded

The Introduction ends with five bullets. Here they are with translations:

What was assumed (and should be questioned)

Every experiment rests on assumptions. Being able to name a paper's assumptions is the mark of a good reader. Here are this one's:

The authors needed problems with four properties. Their exact list:

1. Fine-grained control over complexity — a dial you can turn one notch at a time.
2. Avoid contamination — not sitting in benchmark repositories in the exact form tested.
3. Require only the explicitly provided rules — no outside world knowledge, so you're measuring reasoning and nothing else.
4. Support rigorous, simulator-based evaluation — a program can check every step mechanically.

Puzzle 1 · Tower of Hanoi the star of the paper

Puzzle 2 · Checker Jumping

Puzzle 3 · River Crossing the surprising one

Setup. N actors (written a₁, a₂, … a_N) and their N corresponding agents (A₁, A₂, … A_N). All 2N people start on the left bank with a boat. All must reach the right bank.

The three constraints (quoted structure from Appendix A.1.3):

1. Boat Capacity: the boat carries at most k people. The paper uses k = 2 for N = 2 and N = 3, and k = 3 for larger N.
2. Non-Empty Boat: the boat cannot cross empty — someone must row it.
3. Safety Constraint: an actor may never be in the presence of another actor's agent unless their own agent is also present. This applies on both banks and in the boat. (The story: agents will poach unrepresented clients.)

Puzzle 4 · Blocks World

Setup. Several stacks of lettered blocks (A, B, C, …). An initial configuration and a goal configuration.

The two rules:

1. Top Block Movement: only the topmost block of a stack may be moved.
2. Valid Placement: a block may be placed on an empty position or on top of another block.

The specific design used. The initial state splits the N blocks across two stacks in alphabetical order with a third stack empty as workspace. The goal consolidates everything onto the first stack in an interleaved pattern that alternates between the two original stacks — deliberately requiring complete disassembly and reassembly. The paper's worked examples:

N = 4:  initial [["A","B"], ["C","D"], []]   →  goal [["D","B","C","A"], [], []]
N = 6:  initial [["A","B","C"], ["D","E","F"], []]
        →  goal [["F","C","E","B","D","A"], [], []]

Minimum solution length grows approximately linearly with N. The paper's own difficulty bands: N = 2–7 tests basic planning, N = 8–20 needs longer planning horizons, N > 20 "challenge[s] the limits of sequential reasoning capabilities." They tested up to N = 50.

Side-by-side comparison

The paper's central variable is "complexity". You need to understand three growth rates to read any of its charts. No maths beyond exponents required.

Three ways a number can grow

Compositional depth — the paper's precise definition of difficulty

Why introduce a second word for it? Because N is not comparable across puzzles. N=5 means 31 moves in Hanoi, 35 in Checker Jumping, but only a handful in River Crossing. To compare a Hanoi failure with a River Crossing failure, you need a common currency. Compositional depth is that currency: how many correct steps must be chained, in order, without a single error.

Reading Figure 9 step by step

1. What am I looking at? A scatter plot — dots, not lines — showing how many moves each puzzle needs at each problem size. This is a chart about the puzzles, not about the models. No AI appears in it.
2. x-axis: Problem Size (N) — 1 through 6.
3. y-axis: Compositional Depth (# of Moves) — 0 up to about 65.
4. Colours/shapes: one per puzzle. Yellow squares = Blocks World, orange triangles = Checker Jumping, blue down-triangles = River Crossing, green circles = Tower of Hanoi.
5. What to notice first: at N=1–2 all four puzzles sit essentially on top of each other, near the bottom. They fan apart dramatically as N grows.
6. The pattern: Tower of Hanoi (green) rockets to ~63 at N=6 — that's 2⁶−1. Checker Jumping (orange) reaches ~48 — that's 7²−1. River Crossing and Blocks World stay low and rise gently.
7. Why it matters: it proves the difficulty dial does what the authors claim, and it establishes that the same N means wildly different amounts of work in different puzzles. Without this figure you'd be tempted to compare "N=5 in Hanoi" with "N=5 in River Crossing" as if they were equally hard. They aren't — not remotely.
8. Conclusion to draw: compositional depth, not N, is the fair cross-puzzle measure of length. Whether it's a fair measure of difficulty is the question Figure 10 blows open.

This chapter covers the parts most readers skim. Don't — the credibility of every result rests here, and knowing this material is what separates "I read the paper" from "I understand the paper."

Step 1: The prompt

Models get a system prompt (the standing instructions and rules) and a user prompt (the specific instance). Here is the actual Tower of Hanoi system prompt from Appendix A.1.1, verbatim in substance:

You are a helpful assistant. Solve this puzzle for me.
There are three pegs and n disks of different sizes stacked on the
first peg. The disks are numbered from 1 (smallest) to n (largest).
Disk moves in this puzzle should follow:
  1. Only one disk can be moved at a time.
  2. Each move consists of taking the upper disk from one stack and
     placing it on top of another stack.
  3. A larger disk may not be placed on top of a smaller disk.
The goal is to move the entire stack to the third peg.

Example: With 3 disks numbered 1 (smallest), 2, and 3 (largest), the
initial state is [[3, 2, 1], [], []], and a solution might be:
moves = [[1, 0, 2], [2, 0, 1], [1, 2, 1], [3, 0, 2],
         [1, 1, 0], [2, 1, 2], [1, 0, 2]]

Requirements:
  * When exploring potential solutions in your thinking process,
    always include the corresponding complete list of moves.
  * The positions are 0-indexed (the leftmost peg is 0).
  * Ensure your final answer includes the complete list of moves in
    the format: moves = [[disk id, from peg, to peg], ...]

Step 2: The simulator — mechanical, unarguable grading

Each puzzle has a purpose-built program that plays out a proposed solution step by step. The Tower of Hanoi simulator performs four-layer validation on every single move:

Only if all four pass does the simulator apply the move and update state. After the last move it checks: does the final configuration equal the goal? Both conditions — every move legal and goal reached — are required for success.

Step 3: Extraction — pulling solutions out of the thoughts

To analyse what happens inside the thinking, the authors built an extraction pipeline (Appendix A.2). It:

1. Uses regular expressions — pattern-matching rules for text — to find candidate solutions in both the final answer and the thinking trace. It looks for explicit moves = patterns and also alternative bracket-based structures.
2. Cleans each candidate: strips comments (anything after #) and normalises formatting to a consistent structure.
3. Validates the format, discarding text that merely looks bracket-ish but isn't a move list.
4. Records the token position of each extracted solution within the trace, using the cl100k_base tokeniser. Positions are then normalised by total thought length — converted to a 0-to-1 scale — so a solution found 3,000 tokens into a 6,000-token trace and one found 10,000 tokens into a 20,000-token trace both register as 0.5.
5. Deduplicates: identical move lists are collapsed; only the first occurrence is kept.

Step 4: The full experimental configuration

Methodology: alternatives the authors could have chosen

Figure 1 — the paper's entire argument on one page

Figure 1 sits on page 2 and has four parts. It's the abstract in pictures. If you understand Figure 1 you understand the paper.

Figure 1 (top) — the experimental setup diagram

Figure 1 (bottom left) — accuracy collapses

Figure 1 (bottom middle) — the effort paradox

Figure 1 (bottom right) — where the answer appears in the thinking

Reading bottom-left, step by step

1. x-axis: Complexity (number of disks), 1 to 20. Remember: each step right is a doubling of required moves.
2. y-axis: Accuracy (%), 0 to 100. Higher is better. This is the fraction of the 25 samples that produced a fully correct solution.
3. Two lines: solid blue = thinking model. Dashed pink = the same model with thinking off.
4. Coloured background bands: yellow = the low-complexity regime, blue = medium, pink/red = high. These bands are the paper's "three regimes", and they recur throughout.
5. Shape: both start at 100%. The pink (non-thinking) line falls off a cliff almost immediately — near zero by N=4–5. The blue (thinking) line holds near 100% longer, then plunges between N=7 and N=10. Both are flat at exactly zero from about N=10 onward.
6. First thing to notice: in the yellow band, look carefully — the pink line is at or slightly above blue. At the very easiest problems, thinking is not an advantage.
7. Second thing: the collapse is a cliff, not a slope. It doesn't decay to 40%, then 25%, then 12%. It goes to zero and stays there, flat, for the remaining half of the x-axis.
8. Conclusion: thinking delays the collapse by roughly 3–5 disks. It does not prevent it. In the paper's words: "while thinking models delay this collapse, they also ultimately encounter the same fundamental limitations as their non-thinking counterparts."

Reading bottom-middle — the counterintuitive one

1. y-axis has changed: Response Length (Tokens), 0 to ~20,000. This is effort, not correctness. Higher = the model wrote more.
2. Shape of the blue (thinking) line: it climbs steadily from near zero, peaks around N = 9–10 at roughly 20,000 tokens — then declines, falling to about 12,000 by N=20.
3. Where's the peak relative to the collapse? Compare with bottom-left: accuracy hits zero at about N=10. The token peak is at about N=9–10. Effort peaks right where accuracy dies, then both fall together.
4. Why this is the alarming panel: the model still has 44,000 unused tokens of its 64,000 budget. It is not running out of room. Faced with the hardest problems, it produces less reasoning.

Reading bottom-right — the anatomy of a thought

1. y-axis: "Position within Thoughts", 0 to 1. This is the normalised position from Chapter 11 — 0 means "at the very beginning of the thinking", 1 means "at the very end".
2. x-axis: complexity (disks), 1 to 15.
3. Two marker sets: circles = correct solutions found in the trace; diamonds = incorrect solutions.
4. Left side (simple, N=1–3): the correct-solution markers sit low (early), and incorrect markers sit higher (later). The model finds the right answer first and then keeps going, generating wrong alternatives. That's the "overthinking" phenomenon.
5. Middle (N=4–7): the pattern flips. Correct markers rise toward the end; incorrect ones sit earlier. The model explores wrong paths first and arrives at the correct answer late. This is what productive search looks like.
6. Right side (N≥8): only incorrect markers remain. The correct answer never appears anywhere in the thinking at all.
7. Conclusion: the failure mode changes qualitatively with complexity. It is not one failure getting worse — it is three different behaviours.

Figure 3 — the four puzzles illustrated

Figure 4 — the collapse, in all four puzzles

This is Figure 1's bottom-left, replicated across every puzzle and both model families. It is the paper's main accuracy result.

What to notice, in order

1. Every single panel ends at zero. Eight panels, four puzzles, two model families, two thinking conditions. Not one shows a model surviving to the right-hand edge. This universality is the strongest thing in the paper.
2. The thinking line is generally to the right of the non-thinking line. Thinking buys extra complexity — a few disks, a few checkers — before the same fate.
3. The x-axis scales differ wildly. Blocks World runs to 50; Tower of Hanoi and River Crossing to 20. Don't compare panels by eye position — compare by shape. This is the most common beginner error in reading multi-panel figures.
4. River Crossing collapses at tiny N. Look at the right-hand column: everything is dead by about N=3–4. Yet Tower of Hanoi survives to N=7–8, where the required move count is vastly larger. Remember this — it's the seed of Chapter 15's biggest finding.
5. Blocks World is jagged, especially for DeepSeek-R1, with visible spikes and dips rather than a smooth decline. Real experimental data is noisy; smooth curves in papers should make you slightly suspicious, not reassured.

Figure 5 — the three regimes, at matched compute

Figure 4 asked "who is more accurate?". Figure 5 asks the fairer engineering question: "given the same compute budget, who does better?" — using pass@k from Chapter 7.

Panel-by-panel reading

Finding 1 · The three regimes of complexity

This is the paper's most immediately useful result — the one you can act on tomorrow.

Finding 2 · Complete collapse, not gradual decay

The word "collapse" is doing precise work. Compare two hypothetical decay curves:

Finding 3 · The effort paradox the sign-flip

Now the headline result, in full detail. Figure 6 shows accuracy and thinking-token usage stacked vertically, for five reasoning models across all four puzzles.

The core observation, stated precisely

Unpacking every clause

Figure 13 — the same result with every individual sample shown

Figure 13 in the appendix is Figure 6's bottom row, expanded: three models × four puzzles, with every one of the 25 samples per point plotted individually — green dots for correct solutions, red crosses for incorrect ones, and a blue line for the average.

Why might the effort drop happen? (Careful: this is interpretation)

Section 4.3 is where the paper does something genuinely new. Everything so far measured outcomes. Now they open the box.

The method, recapped

For every intermediate solution found inside a Claude-3.7-Sonnet-Thinking trace, they recorded three things:

1. Its relative position in the reasoning process (normalised 0 to 1 by total thought length).
2. Its correctness, as validated by the puzzle simulator — not by the model's own opinion of it.
3. The complexity of the problem it belonged to.

Then they plotted position against complexity, colouring by correctness.

Reading Figure 7a — this takes care, so go slowly

1. The y-axis is not accuracy. This trips up everyone. It is position in time within the reasoning. A marker at 0.2 means "this candidate solution appeared one-fifth of the way through the model's thinking."
2. The shaded blobs are distribution densities — wider means more solutions occurred at that height. Think of them as a histogram turned sideways and smoothed.
3. Left side of each panel (simple problems): green (correct) markers sit lower than pink (incorrect) markers. The model finds the right answer early, then spends the rest of its thinking generating wrong alternatives.
4. Middle (moderate problems): the two swap. Pink sits lower, green sits higher. The model explores wrong ideas first and reaches the correct one late.
5. Right side (high complexity): green markers disappear entirely. Only pink remains. "The model fails to generate any correct solutions within the thought."

Figure 7b — accuracy within the thinking, over time

Reading Figure 7b

• N=1–3 (yellow/olive lines): start high — sometimes at 100% — and decline or oscillate as thinking continues. The model's later ideas are worse than its first ones. More thinking actively hurts. This is overthinking measured directly.
• N=4–7 (blue lines): start low and improve with continued thinking. Search is paying off. This is the regime where "let it think longer" is genuinely good advice.
• N≥8 (flat lines at the bottom): hug zero across the entire trace. No amount of additional thinking produces anything correct.

Section 4.4 of the paper is titled "Open Questions: Puzzling Behavior of Reasoning Models." The authors are candid: these are results they found surprising and cannot fully explain. That candour is a feature.

Surprise 1 · Giving the model the algorithm doesn't help

This is the cleanest, most decisive experiment in the paper. Setup: take the Tower of Hanoi prompt and append the complete recursive solution algorithm as pseudocode. The model no longer needs to discover anything — it only needs to execute. Here is the algorithm they provided, verbatim:

ALGORITHM Solve(n, source, target, auxiliary, moves)
    // n = number of disks to move
    // source = starting peg (0, 1, or 2)
    // target = destination peg (0, 1, or 2)
    // auxiliary = the unused peg (0, 1, or 2)
    // moves = list to store the sequence of moves

    IF n equals 1 THEN
        // Get the top disk from source peg
        disk = the top disk on the source peg
        // Add the move to our list: [disk_id, source, target]
        ADD [disk, source, target] to moves
        RETURN
    END IF

    // Move n-1 disks from source to auxiliary peg
    Solve(n-1, source, auxiliary, target, moves)

    // Move the nth disk from source to target
    disk = the top disk on the source peg
    ADD [disk, source, target] to moves

    // Move n-1 disks from auxiliary to target
    Solve(n-1, auxiliary, target, source, moves)
END ALGORITHM

Surprise 2 · Depth doesn't predict difficulty across puzzles

Recall Figure 9: it established compositional depth as a ruler. Figure 10 now asks whether that ruler predicts anything — plotting accuracy directly against compositional depth for all four puzzles at once.

What you should notice, and it should bother you

1. Within a single puzzle type (follow just the green circles, say) the expected relationship holds: as depth increases, accuracy decreases. Sensible.
2. Across puzzle types, the relationship shatters. Look at the far right of each panel: green circles (Tower of Hanoi) sit high — above 50% — at depths around 10² = 100 moves. Now look at the far left: blue triangles (River Crossing) sit at zero at depths around 10¹ = 10 moves.
3. Stated plainly: models solve 100-move Tower of Hanoi problems while failing 10-move River Crossing problems. A tenfold shorter problem, and they can't do it.

The authors' own explanation

In Section 4.4 they offer a hypothesis, and they are careful to mark it as one: "This likely suggests that examples of River Crossing with N>2 are scarce on the web, meaning LRMs may not have frequently encountered or memorized such instances during training."

Surprise 3 · Where models make their first mistake

Figures 8c/8d, 11 and 12 all examine the first failure move: at which step in the proposed sequence does the first illegal or wrong move occur? This is a much finer-grained signal than pass/fail.

Figures 11 and 12 — failure positions in detail

Figure 11 reveals something the paper flags as "non-monotonic failure behavior" — a phrase worth unpacking. "Monotonic" means "always moving in one direction". Non-monotonic here means the failure position goes up and down unpredictably as complexity rises.

Figure 11 contains one further oddity the authors highlight: in the fully-collapsed regime (Tower of Hanoi N≥15, Blocks World N≥40), non-thinking models sometimes fail later in the sequence than thinking models do. The model with the reasoning mechanism does worse on this measure. The paper's reading: "compositional reasoning failures in LLMs are not simply due to insufficient context length or inference compute, but rather reflect fundamental limitations in how models maintain algorithmic consistency across problem scales."

The exact numbers the paper reports

The conclusion, in the authors' own words

The five findings, consolidated

What surprised the researchers

Worth listing explicitly — a paper's surprises tell you where the field's mental model was wrong.

The limitations, as the authors state them

Quoted and translated. Read this section of any paper you care about — it's where authors are most honest.

Practical implications — what to actually do differently

Future work the paper points toward

• Understanding symbolic manipulation. The paper explicitly calls for it, citing Marcus [44] and GSM-Symbolic [6]: "further research is needed to understand the symbolic manipulation capabilities of such models."
• Why does effort decline? The mechanism behind the scaling limit is unexplained and would require white-box access.
• Better self-verification. If a model could recognise its own correct answers, overthinking would vanish and search would terminate properly.
• Hybrid neuro-symbolic architectures. The implicit suggestion throughout: pair the pattern-recognition strengths of neural models with exact symbolic execution.
• Beyond puzzles. Do these thresholds appear in knowledge-intensive domains where validation is fuzzy? The authors flag this as unknown.

Objection 1 · The token-budget ceiling the strongest

This is the most substantive criticism and it deserves your full attention. Let's build it from arithmetic you can check.

The argument: at N=15, the optimal solution is 32,767 moves. Written out at ~10 tokens each, that's over 300,000 tokens — roughly five times the entire 64,000-token budget. The model physically cannot emit a correct answer, no matter how perfectly it reasons. So the reported 0% accuracy at those complexities may measure an output-length constraint rather than a reasoning failure.

This is the core of a widely-circulated rebuttal titled "The Illusion of the Illusion of Thinking" (C. Opus & A. Lawsen, 2025), which also reported finding text in some model traces where the model explicitly notes it will not enumerate every move because the sequence is too long — behaving sensibly, and being graded as a failure.

Objection 2 · Some puzzle instances may be unsolvable

Recall the River Crossing setup: boat capacity k = 3 for N > 3 pairs. It is a known result in the planning literature that the classic missionaries-and-cannibals-style constraint structure becomes unsolvable for larger numbers of pairs with small boat capacity — with capacity 3, the puzzle has no solution beyond a certain number of pairs.

If some tested instances had no valid solution at all, then 0% accuracy on those instances is guaranteed by mathematics, not by model limitation. A perfect reasoner would also score 0%.

Objection 3 · Is "write out 1,023 moves" a fair test of reasoning?

A person who understands Tower of Hanoi perfectly would never hand-write 1,023 moves. They'd write four lines of code. Requiring full enumeration arguably tests transcription stamina alongside reasoning, and conflates two very different abilities.

The counter-argument, which is strong: the paper isn't claiming to measure abstract understanding. It's measuring the ability to execute exact multi-step procedures reliably — which is a capability the field explicitly claims for these models and which matters enormously for agentic deployments. And the algorithm-given experiment specifically isolates execution from understanding. Still, when you report the finding, say "cannot reliably execute long exact procedures", not "does not understand the puzzle". They're different claims and only the first is supported.

Objection 4 · Output format vs capability

Models were required to produce a specific bracket format. Some failures could be formatting or parsing artefacts rather than reasoning errors. Mitigating factors: the extraction pipeline explicitly handled multiple formats including "alternative bracket-based solutions", the prompts included worked examples, and formatting problems wouldn't produce a sharp complexity-dependent cliff — they'd produce roughly constant noise across all N. A cliff needs a cliff-shaped cause.

Objection 5 · Would newer models change the picture?

The paper tested models available in early 2025. Reasoning models have continued to improve. It is entirely plausible that today's models have later collapse thresholds.

The steelman on each side

The one-page cheat sheet

Numbers worth memorising

Flashcards

Click a card to flip. Use the buttons or ← / → keys to move through the deck.

Frequently asked questions

Glossary

Every technical term used in this course, defined. Terms marked with a dotted underline elsewhere on this page link here.

Ablation

Deliberately removing one component of a system to test whether it mattered. Comparing thinking vs non-thinking models is effectively an ablation of the thinking mechanism.

arXiv preprint

A paper posted publicly before peer review at arxiv.org. Fast and free, but unvetted. This paper is one.

Attention

The Transformer mechanism by which, when processing any token, the model weighs how much every earlier token should influence it. The core innovation that made modern LLMs work.

Benchmark

A standardised shared test used to compare systems. Examples here: MATH-500, AIME24, AIME25.

Black-box access

Being able to send inputs and read outputs, but not inspect internal states. A limitation the authors acknowledge for the closed models they tested.

Chain of Thought (CoT)

Prompting a model to write intermediate reasoning steps before its final answer. Improves multi-step performance, but the written steps are not guaranteed to describe the model's actual process.

cl100k_base

The specific tokeniser used to count token positions consistently across all models in this paper.

Collapse

Accuracy dropping to exactly zero past a complexity threshold, rather than declining gradually. The shape matters: cliffs suggest missing capability, slopes suggest strained capability.

Compositional depth

The number of sequential operations (moves) required for a full solution. The paper's cross-puzzle common currency for problem length.

Data contamination

Test questions appearing in a model's training data, so a high score may reflect memorisation rather than capability.

Embedding

The list of numbers representing a token inside the model. How text becomes something a network can compute on.

Frontier model

Informal term for the most capable models currently available. In this paper: Claude 3.7, DeepSeek-R1, o3-mini.

Generation limit

The maximum number of tokens a model is permitted to produce for one response. Here, 64,000 for all models.

Gradient descent

The training algorithm that adjusts parameters in whichever direction reduces loss. Intuitively: rolling downhill on an error landscape.

Inference compute

The amount of computation spent producing one answer, measured in this paper by counting generated tokens.

Inference

Running a trained model to produce output. Parameters are frozen. Everything in this paper happens at inference time.

Large Language Model (LLM)

A model trained to predict the next token in text, producing all its capabilities from that single operation.

Large Reasoning Model (LRM)

An LLM trained (largely via reinforcement learning) to generate a long self-reflective thinking process before answering. Examples: o1/o3, DeepSeek-R1, Claude 3.7 Sonnet Thinking.

Logarithmic scale

An axis where equal distances represent equal multiplications (1, 10, 100, 1000) rather than equal additions. Used when data spans orders of magnitude.

Loss

A single number quantifying how wrong a prediction was during training. Lower is better; training minimises it.

Monotonic / non-monotonic

Monotonic means consistently moving in one direction. The paper reports non-monotonic failure behaviour: failure positions rise and fall unpredictably with complexity, indicating no consistent procedure is being executed.

Overthinking

Finding a correct solution early in the reasoning trace and then continuing to generate — mostly incorrect — alternatives. Symptomatic of weak self-verification.

Parameters (weights)

The billions of adjustable numbers inside a model, set during training. All of the model's knowledge lives here.

pass@k

The probability that at least one of k independent attempts is correct. Measures capability ceiling rather than reliability. Formula: 1 − (1 − p)^k for independent attempts.

Pattern matching

Producing outputs resembling solutions seen during training rather than deriving them. Diagnostic symptom: collapse on structurally unfamiliar problems even when they're logically simpler.

Peer review

Independent experts vetting a paper before publication. Quality control that preprints have not yet undergone.

Pre-training

The initial, expensive training phase where a model learns next-token prediction over vast text.

Regular expression (regex)

A pattern-matching rule for text. Used in the paper's extraction pipeline to find candidate move-lists inside traces.

Reinforcement learning (RL)

Training by trial and reward rather than imitation. For reasoning models the reward is typically "was the final answer verifiably correct?" — from which self-reflection and backtracking emerged.

Regimes (three)

Low complexity: non-thinking models match or beat thinking models more cheaply. Medium: thinking models win. High: both collapse to zero regardless of compute.

Simulator

A deterministic program that replays a proposed solution move by move, validating each against the puzzle's constraints and checking whether the goal state is reached.

State of the art (SOTA)

The best published performance on a benchmark at a given time.

Temperature

A setting controlling randomness in token selection. 0 = always take the most likely token; 1.0 = sample proportionally to probabilities. This paper used 1.0.

Thinking tokens

The tokens a reasoning model generates inside its private reasoning section before producing its final answer. The paper's proxy for reasoning effort.

Token

A chunk of text — typically a word or word-fragment — that models read and write one at a time. Roughly 1 token ≈ 0.75 English words.

Transformer

The neural network architecture underlying all modern LLMs, built around the attention mechanism.

Final exam

Ten questions spanning the whole course. No hints. Aim for 8+.

Self-assessment: can you teach it?

Tick each only when you can say it out loud, unaided, to an imaginary colleague. Your progress is saved as you go.

Final revision notes — the story in ten sentences