principles.fyi · the brain · concept

DPO (direct preference optimization)

Aligns straight from preference pairs by favouring chosen over rejected, with no separate reward model.

L_DPO = -log sigmoid( beta*log(pi(o_w)/pi_ref(o_w)) - beta*log(pi(o_l)/pi_ref(o_l)) )

DPO skips RLHF's two-stage setup: instead of training a reward model and then doing RL, it finetunes directly on preference pairs by raising the policy-versus-reference log-probability ratio for the chosen answer relative to the rejected one. The math works out because, under the same KL-regularized objective RLHF optimizes, the reward can be rewritten in terms of that log-ratio and the awkward partition function cancels — so there is no reward model to train and no sampling needed. That makes it simpler and more stable than RLHF while targeting the same preference-alignment goal.