The Math Beneath · Part 7 / 8

The cost of wrong beliefs

Believe q while the world draws from p, and your average surprise splits in two — the world's own entropy, a floor no model can beat, plus KL, the penalty for being wrong. That's the training loss.

Last chapter’s world was honest: the probabilities weighting the average were the same ones generating the days. But a model doesn’t get that luxury. The world draws from p. You believe q.

Your surprise at each outcome is priced by your beliefs — log(1/q). How often each outcome actually arrives is set by the world — p. So your average surprise is:

H(p, q) = Σ p · log(1/q)

This is cross-entropy: expected surprise, experienced through the wrong beliefs.

The floor and the gap

Subtract the honest world’s entropy from it and watch it split:

H(p, q) = H(p) + KL(p ‖ q)

Two parts, with completely different characters:

• H(p) — the floor. The world’s own unpredictability. If tomorrow is genuinely 50/50, no model ever built will average under 1 bit of surprise. Not wrongness — weather.
• KL(p ‖ q) — the gap. The extra surprise you inflict on yourself by believing q instead of p. KL divergence is never negative, and it hits zero in exactly one place: q = p.

So “be a good model” has a precise meaning: you cannot lower the floor; you can only close the gap.

Try it: press match the world — the amber penalty vanishes and only the floor remains. Then confident & wrong — cross-entropy explodes, because paying log(1/q) with a tiny q on an outcome that actually happens is catastrophically expensive. Note that even confident & right carries a penalty: the world says 70% Paris, and claiming 97% is also a wrong belief.

This is the loss

You have now assembled, from scratch, the objective under nearly everything in this library:

• Pretraining a transformer minimizes cross-entropy against the true next word — become less surprised by real text; the irreducible entropy of language itself is the floor.
• Training the reward model and DPO minimize cross-entropy on preference verdicts — the same loss, because chapter three showed they’re logistic regression underneath.
• RLHF uses KL directly as a leash: stray from the reference model’s distribution and the divergence itself becomes a cost.

The loss is now a number. The last chapter is about the only thing left: which way to move to make it smaller.