Grading something that does things

Grading a translation or a summary means arguing about quality — there is no single right answer to compare against, only better and worse.

Code is different. You can run it.

And you have to, because the surface tells you nothing: a recursive factorial and an iterative one share almost no text and are equally correct. Any text-similarity score marks one of them wrong. Execution does not care what it looks like.

One number, two questions

So run it against tests. Except the model does not answer the same way twice, which means “did it work?” is ambiguous in a way that matters.

pass@k is the chance that at least one of k samples passes. And the two ends of it are different jobs:

pass@1 — can I trust it once? Wants careful, low-temperature sampling: the single best guess.
pass@k — can I find a working answer at all? Wants adventurous sampling: k different attempts beat k copies of the same one, when a test suite sorts them out afterwards.

Reporting one number for a system hides which question was asked. It also means the same model can look very different depending on how you prompted and sampled it — on GSM8K, the same weights went from about 5% with a bare prompt to 21% with chain-of-thought to 55–69% with sampling plus a verifier. You are never measuring a model alone; you are measuring a model plus how you got it to try.

Verifying beats generating

That last jump is worth pulling out, because it is a design principle rather than a benchmark result.

Generate a hundred candidates, train a model to score them, return the best. On GSM8K this bought gains comparable to a roughly thirty-fold increase in model size. Improving the judge beat improving the writer.

It generalises: wherever answers are cheap to check and expensive to get right, spend on checking. Tests, type checkers, linters, a second model reading the diff. This is process supervision again, now as an architecture rather than a training signal.

Where the benchmark stops resembling the job

HumanEval asks for one function from a docstring. Frontier models pass over 90% of it, and it is effectively finished as a measurement.

SWE-bench asks for something else: here is a real GitHub issue and a real repository — produce a patch that makes the tests pass. Thousands of files, no pointer to the relevant one.

Early scores were around 2%. The gap between writing a function and changing a codebase turned out to be enormous, and high HumanEval scores did not predict it at all. What closed it was not bigger models on their own — it was wrapping them in the loop from chapter one: explore, run the tests, read the failure, try again. By 2025 the best systems were over 60%.

That is the clearest evidence in this book that the harness is part of the system, not packaging around it.

Go deeper: passing tests is not the same as correct

An audit of SWE-bench (SWE-bench+) found roughly 11% of “passing” patches were suspect — satisfying what the tests checked without genuinely fixing the issue. Sometimes the fix was leaked in the issue text; sometimes the tests were simply weak.

This is the ceiling on execution-based grading, and it is not fixable by better execution. Tests are a sample of correctness, not correctness. An agent optimising against them is doing exactly what the goal-satisfaction problem predicts — which is why the honest setup keeps some held-out tests the agent never sees.

That is the loop, end to end. Now change the tokens: nothing in any of this said they had to be words.

Sources · 5

Full bibliography →