policy

The LLM itself, viewed as the thing choosing each next token.

pi(next token | tokens so far)

In the reinforcement-learning framing of RLHF, the policy (written pi) is just the language model: given the tokens so far, it produces a distribution over the next token and picks from it, so generating an answer is a sequence of choices. RLHF tunes this policy to produce outputs the reward model scores highly, while a KL penalty keeps it from straying too far from the frozen reference policy it started as. So policy is simply the RL name for the model whose weights are being updated.

Appears in

Nudge it with the critic, on a leash Post-training · pt 5
Delete the critic Post-training · pt 6

Nearby in the brain

KL divergence reference policy reward model RLHF PPO reward reward hacking Bradley-Terry model