reference policy

The frozen pre-RL model the tuned model is kept close to during alignment.

penalty pulls pi toward pi_ref (pi_ref frozen)

The reference policy (pi_ref) is a snapshot of the model taken before reinforcement tuning begins, with its weights frozen. The KL-divergence penalty in RLHF (and the log-ratio in DPO) measures the tuned policy against this anchor, so it acts as the fixed point on the other end of the KL leash. Keeping the policy near it preserves the fluency and knowledge built up in pretraining and instruction tuning, preventing alignment from degrading the model while it chases reward.

Appears in

Nudge it with the critic, on a leash Post-training · pt 5
Delete the critic Post-training · pt 6

Nearby in the brain

KL divergence policy reward model RLHF PPO reward reward hacking Bradley-Terry model