Post-training · Part 7 / 7

Think longer at answer-time

Even after all the training, you can spend more compute when the model answers — let it write its reasoning steps, and hard problems start coming out right.

So far the whole story has been about training — spending compute ahead of time to shape the model.

Pretraining taught it to predict the likely next word.
taught it to follow what you ask.
taught it to lean toward answers people prefer.

All of that is baked in before you ever type a prompt. But there is one more lever, and it isn’t training at all.

Spend compute at answer-time

When the trained model finally answers, it runs forward through its weights and produces words. You can choose to spend more computation in that moment — let it generate more before it commits to a final answer. That class of methods is : extra thinking at answer-time, no weights changed.

Why would more words help? A single forward pass tries to guess the answer in one leap. For an easy question that’s fine. For a hard one, leaping straight to the end is exactly where it goes wrong.

Let it write the steps

People don’t solve a hard problem in one jump either — they break it into steps. So nudge the model to do the same, right in the prompt. That is prompting (Wei et al., 2022), and the technique is almost embarrassingly simple: in a few-shot prompt, show worked examples that include the reasoning steps, not just the answer. The model then writes step-by-step reasoning for your new problem too — and writing the steps is what carries it to the right answer.

Watch the same math word problem both ways:

Straight to the answer: “The answer is 27.” — wrong.

With steps: “The cafeteria had 23 apples. They used 20 for lunch, so 23 − 20 = 3. They bought 6 more, so 3 + 6 = 9. The answer is 9.” — right.

Try it: run the hard problem with steps turned off, then on. The model’s weights never change — only how much it writes before answering. That extra writing is the extra compute.

Nothing in the model got smarter. It just stopped trying to do all the work in one move. Each step is an easier prediction than the whole leap, and the steps compose into the final answer. On hard reasoning tasks the difference is large — grade-school math word problems (GSM8k) and the hardest BIG-Bench tasks both jump once the model is allowed to think out loud.

Why steps beat the leap

The model only ever does one thing: predict the next token. A one-shot answer asks it to map the whole problem to the final number in a single pass — a hard, brittle function to get right. Writing steps turns that into a chain of small predictions, each conditioned on the words already written. The intermediate text becomes scratch work the model can read back, so later tokens build on earlier results instead of re-deriving everything at once. More tokens written = more compute spent = a longer chain of easy predictions standing in for one impossible leap.

The lesson is its own principle: past some point, you stop improving the model and start giving it room to think. Likely became good through training; here, good gets better still by simply thinking longer.

Sources · 4

Full bibliography →