preference alignment

Aligning a model using judgments of which of two answers is better.

prompt + (chosen vs rejected) -> tune the model toward chosen

Preference alignment is the part of post-training that uses comparative human judgments — given a prompt and two candidate answers, which one is preferred — to shape the model's behaviour. Rather than telling the model the single right answer, it teaches the model the relative ranking, which captures fuzzy qualities like helpfulness and tone that are hard to write down as one correct target. The two main ways to do it are RLHF, which trains a reward model and then reinforcement-learns against it, and DPO, which optimizes the preference pairs directly without a separate reward model.

Appears in

Likely is not good Post-training · pt 1
Judging beats writing Post-training · pt 3
Think longer at answer-time Post-training · pt 7

Nearby in the brain

instruction tuning base model model alignment post-training preference data reward sycophancy chain of thought