Comparing reasoning paths

Writing the steps gives later predictions access to earlier working. A single sampled path can still follow an unhelpful opening for too long.

It can notice a mistake and write a correction. What ordinary sampling does not provide by itself is a separate record of alternative branches to compare and revisit. Explicit search adds that record.

Two ways out

Run it several times. Sample several reasoning paths and compare their final answers. Taking the most common answer is self-consistency. It can help when correct answers recur more often than any particular wrong answer. If the samples share a systematic mistake, a majority vote can repeat it.

It has a limit: every chain still has to get all the way to an answer before you learn anything. Five chains, five complete failures, five full costs.

Or branch, and judge before the end. Generate a few candidate next moves. Score each one. Keep the promising, drop the hopeless, and expand the best. When a branch dies, go back up and take another — tree of thoughts.

On the Game of 24, this is the difference between roughly 4% and roughly 74% for the same model.

The part that actually does the work

Look at what the score in the widget is attached to. Not an answer — there is no answer yet. It is attached to a leftover bag of numbers: sure, maybe, hopeless.

That distinction has a name and it matters more than the tree does.

Outcome supervision grades the final answer. One bit, at the very end, for the whole chain — and a model can earn it with reasoning that was wrong in ways that happened to cancel.

Process supervision grades each step as it is made. Feedback everywhere, and the credit lands on the step that deserved it.

Train a model to do that grading and you have a process reward model — OpenAI’s PRM800K collected around 800,000 human judgements of individual step correctness for exactly this. It is useful in both directions: at answer-time it scores branches so the search knows where to look, and at training time it is a far sharper signal than “right or wrong at the end.”

Then stop bolting it on

Everything above is scaffolding around the model — sampling, scoring, backtracking, all written by you.

The obvious next move is to train the model to do it internally. Reward reasoning that reaches correct answers, and let reinforcement learning find whatever thinking produces them. That is a reasoning model: it may think for thirty seconds before its first visible word, and on competition mathematics the jump is not incremental — o1 reported around 83% on AIME 2024 where GPT-4o managed roughly 12%.

The DeepSeek paper distinguishes R1-Zero, which applied reinforcement learning to a pretrained model without a preliminary supervised fine-tuning stage, from R1, which also used cold-start examples and multiple training stages. The authors reported self-checking and other extended reasoning behavior in R1-Zero. This does not mean it learned language or reasoning from no prior training.

Go deeper: what this costs, and when not to pay it

Thinking is billed. A reasoning model can generate thousands of hidden tokens per query, and you pay for tokens you never see. Latency follows: thirty seconds is fine for a hard proof and absurd for what is the capital of France.

The internal chain is also often not shown, which takes away the readability that made the agent loop debuggable in the first place.

So the deployment question is routing, not adoption: match the depth of thinking to the difficulty of the task, and send the easy majority somewhere fast.

Reasoning, tools, memory, search. Now the uncomfortable part: what all of this looks like when it goes wrong.

Sources · 6

Full bibliography →