Post-training · Part 4 / 7

Learn a critic

We have judgments but not the hidden scores. So train a model to produce them — a reward model — by pushing the chosen answer's score above the rejected one's.

The pairing rule needed a hidden score for each answer: P(A beats B) = sigmoid(z_A − z_B). But nobody handed us those zs. All we collected was a pile of — this answer beat that one — one bit per comparison.

So we don’t look the scores up. We learn a function that makes them.

A critic that outputs one number

Take a model that already reads language well — a . Normally its last layer turns the final vector into a probability over the whole vocabulary: a guess at the next word. We don’t want a word. We want a verdict.

So strip that last layer off and bolt on a new one: a single linear layer that collapses the final vector down to one scalar. That number is the . Feed in a prompt and an answer, get back r(x, o) — how good this answer is for this prompt. That whole machine is the .

The new layer starts with random weights — at first the score is noise. We have to teach it.

Push the winner up, the loser down

We already know the shape the scores should obey. Reuse it, but with the learned rewards in place of the hidden ones:

P(o_w beats o_l | x) = sigmoid(r(x, o_w) − r(x, o_l))

For one preference pair — winner o_w, loser o_l — we want that probability to be high. The loss is just the negative log of it:

L = −log sigmoid(r(x, o_w) − r(x, o_l))

This is the same cross-entropy idea that trains the rest of the model: it’s small only when the gap r(x, o_w) − r(x, o_l) is large and positive. Average it over the whole pile of judgments and run gradient descent. Every step widens the gap — nudging the chosen answer’s score up and the rejected one’s down — until the critic’s scores agree with the people who did the rating.

Try it: step the training in the widget. The two bars start tangled; each nudge spreads them apart, and the loss falls as the gap grows.

This is the turned into a teacher: it never needs a “correct score,” only which of two answers won.

Go deeper: why one number is enough

The reward model never has to output a calibrated quality on some absolute 0-to-10 scale. The loss only ever sees a difference of two scores, passed through sigmoid. Add a constant to every reward and nothing changes — the gaps stay put. So the model is free to settle on whatever offset it likes; all that gets pinned down is the ranking. That’s why a single scalar, with no fixed meaning of its own, is all the critic needs to learn.

What a critic is good for

Once you have a scorer, it pays off right away. Best-of-N: sample several answers from a model, score them all, keep the highest. Data cleaning: rate your training examples and toss the low-scoring ones.

But the real prize comes next: we point this critic at the model itself and let its scores reshape what the model writes.

Sources · 6

Full bibliography →