principles.fyi · the brain · concept

RLHF (reinforcement learning from human feedback)

Train a reward model from people's preferences, then tune the model to score high on it without drifting too far.

pi* = argmax E[ r(x,o) - beta * KL(pi || pi_ref) ]

RLHF turns preference data into alignment in two stages: first a reward model learns to score outputs from people's chosen-versus-rejected comparisons, then the LLM (the policy) is reinforcement-tuned to maximize that reward. To stop it from gaming the reward and producing degenerate text, the objective subtracts a beta-weighted KL-divergence penalty that keeps the tuned policy close to a frozen reference policy. The RL update is usually carried out with PPO; DPO is a later shortcut that skips the separate reward model entirely.