Post-training · Part 6 / 7

Delete the critic

A bit of algebra makes the reward model vanish — the policy is its own implicit reward, so you can train straight on preference pairs. That is DPO.

Last chapter’s recipe needed a whole pipeline: train a separate to score answers, then keep sampling fresh answers from the and chasing that score. Four moving models. Slow. Touchy.

What if you never built the scorer at all — and trained the model straight on the preference pairs you already have?

The reward was hiding in the model

Start from the exact same goal as : get high reward, but don’t drift far from the you started from. That goal has a known best answer — and you can flip it inside out. Instead of reward → policy, write reward in terms of the policy:

r(x, o) = β · log( π(o|x) / π_ref(o|x) ) + β · log Z(x)

Read it plainly: a good answer is just one the trained model finds more likely than the reference does. That ratio is the reward. The ugly piece is Z(x) — a sum over every possible answer, far too big to ever compute.

The trick: it cancels

Here’s the gift. Preferences only ever compare two answers — the model cares only about the difference r(x, oᵢ) − r(x, oⱼ). Subtract the two rewards and the β · log Z(x) term, identical in both, simply cancels.

The intractable sum is gone. The chance a human prefers answer i over j becomes nothing but two log-ratios:

P(oᵢ ≻ oⱼ | x) = σ( β·log(π(oᵢ)/π_ref(oᵢ)) − β·log(π(oⱼ)/π_ref(oⱼ)) )

No scorer. No sampling. Just the model and its frozen copy.

Try it: toggle Panel 1 between RLHF and DPO to see the two middle stages vanish. Then in Panel 2 hit DPO step or auto-run — the chosen ratio climbs above the frozen reference while the rejected one sinks below it, and the margin grows while the loss falls.

The whole loss

So trains directly on (winner, loser) pairs. For each pair, push the chosen answer’s log-ratio up and the rejected one’s down:

L = − log σ( β·log(π(o_w)/π_ref(o_w)) − β·log(π(o_l)/π_ref(o_l)) )

That’s ordinary cross-entropy — Book 01’s loss again — now grading which answer won. Gradient descent raises the model’s probability on the preferred answer and lowers it on the rejected one. And because every term is measured against the reference, drifting too far costs probability mass on the reference’s own answers — a built-in leash, no separate penalty needed. (β is small, around 0.1 to 0.01: smaller means a shorter leash.)

The model became its own scorer. Its log-probability ratio to the reference is the reward — no critic to train, ever.

Go deeper: why only two models, not four

PPO-style RLHF juggles four models at once: the policy being trained, a frozen reference for the KL term, the reward model, and a value network the critic uses to estimate advantage. DPO keeps just two — the policy and the frozen reference. The reward model is absorbed into the loss as a log-ratio; there’s no value network because there’s no on-policy rollout to estimate returns for. Two models in memory instead of four is most of why DPO is so much cheaper and steadier to run.

Next: what happens when the leash slips and the model starts gaming the reward instead of earning it.

Sources · 2

Full bibliography →