Post-training · Part 3 / 7

Judging beats writing

People struggle to write the perfect answer but easily judge which of two answers is better. Turn that judgment into a number with one S-shaped curve.

To teach by example, someone has to write the perfect answer.

That’s the catch. Writing one is hard — and even a careful answer can overstate, mislead, or be technically correct yet useless.

You don’t have to write it to know it’s worse

Read these two replies to “Is garlic a good natural antibiotic for colds?”

“It can be helpful against colds, but may make you stink.”
“It might be one of the best natural antibiotics out there, so it would help if you have a cold.”

You can’t easily write the ideal answer. But you know the second one is worse — it’s overconfident, a quiet flavour of over what’s true.

That’s the whole idea. People don’t need to know how to produce a good answer. They just need an opinion about which of two answers is better. Judging is far easier than writing.

So we collect : a prompt, two sampled answers, and a verdict — a chosen (winner) and a rejected (loser). Written o_i > o_j for prompt x. This is the raw material for .

Turn a verdict into a number

A verdict is just this one beats that one. To train on it we need a number.

So assume each answer has a hidden z — how good it is. The better answer should usually win. But “usually” matters: if the two answers are close, the verdict could flip.

The trick is to look only at the difference z_i − z_j and squash it through the S-shaped logistic sigmoid:

P(o_i > o_j | x) = sigmoid(z_i − z_j) = 1 / (1 + e^−(z_i − z_j))

This is the Bradley–Terry model (1952). Read off the three cases:

difference near 0 → probability near 0.5 — a toss-up, no real preference
big positive difference → probability near 1 — o_i almost always wins
big negative difference → probability near 0 — o_j almost always wins

Try it: set both scores equal and watch the probability lock onto 0.5. Now nudge one up a hair — the verdict barely leans. Nudge it far — it becomes near-certain.

The model only ever sees the gap, not the raw scores — which is why preferences map to it so cleanly. Shift both scores by the same amount and nothing changes.

Go deeper: it's just logistic regression in disguise

Start from what a probability implies, not from the sigmoid. The log-odds of an event is log( P / (1 − P) ). Set that equal to the score difference δ = z_i − z_j:

log( P / (1 − P) ) = δ

Exponentiate both sides, then solve for P:

P / (1 − P) = e^δ ⟹ P = e^δ / (1 + e^δ) = 1 / (1 + e^−δ) = sigmoid(δ)

So the sigmoid isn’t a magic curve — it’s the only shape consistent with “the score difference is the log-odds of preferring the winner.” That’s the same δ the logit plays in ordinary binary logistic regression, which is also why training on preferences reuses the familiar cross-entropy loss.

We now have a clean target: a probability we can fit. Next, we hand that hidden score over to a model and let it learn to be the judge.

Sources · 8

Full bibliography →