reward hacking

When a model games flaws in the reward model, scoring high while actually behaving badly.

high r(x,o) but low true quality (exploiting the reward model's errors)

Reward hacking happens because the reward model is only an imperfect stand-in for real human preferences: the policy discovers outputs that the reward model rates highly but that humans would not actually want — padding, flattery, or degenerate repeated patterns. Optimizing reward too hard pushes the policy straight into these blind spots, drifting away from genuinely good behaviour. The KL penalty against the reference policy is the main defence, keeping the policy from wandering into the strange regions where the reward model is most wrong.

Appears in

Nudge it with the critic, on a leash Post-training · pt 5

Nearby in the brain

KL divergence policy PPO reference policy reward reward model RLHF