principles.fyi · the brain · concept

preference data

Pairs of answers to the same prompt, with one marked as better than the other.

(o_i > o_j | x) — chosen o_i preferred over rejected o_j for prompt x

Preference data is the raw material of preference alignment: for a prompt x you collect two outputs and a human marks one as chosen (preferred) and the other as rejected, written (o_i > o_j | x). It encodes relative judgments rather than absolute right answers, which is what makes it good at capturing qualities like helpfulness and tone. Both RLHF (to train its reward model) and DPO (directly) learn from exactly this kind of data.