reward

The single score a reward model assigns to an answer — higher means more preferred.

reward = r(x,o) (scalar; higher = more preferred)

The reward is the scalar number the reward model outputs for a given prompt and output, where a higher value means the answer is judged better. In the RL framing of RLHF it is the signal the policy is trained to increase, standing in for the human preferences the reward model learned from. Because it is only an approximation of true human judgment, blindly maximizing it can lead the policy into reward-hacking, which is why a KL penalty keeps the policy anchored to its reference.

Appears in

Judging beats writing Post-training · pt 3
Learn a critic Post-training · pt 4
Nudge it with the critic, on a leash Post-training · pt 5
Evidence adds points The Math Beneath · pt 3
What to expect The Math Beneath · pt 5
The cost of wrong beliefs The Math Beneath · pt 7

Nearby in the brain

Bradley-Terry model cross-entropy loss KL divergence RLHF preference data reward model odds sigmoid