principles.fyi · the brain · concept

entropy

Expected surprise: how surprising a distribution is on an average draw.

H(p) = Σ p · log₂(1/p)

Entropy H(p) weights each outcome's surprise log(1/p) by its probability, giving how hard the distribution is to predict overall — zero for a certain world, maximal when every outcome is equally likely. A rare disaster contributes little despite enormous surprise, because it almost never happens. Shannon's coding theorem gives the same number a second meaning: the minimum average bits needed to describe draws from the source, which is why predicting well and compressing well are the same skill. It is also the unbeatable floor inside cross-entropy — the part of a model's loss that comes from the world itself being uncertain.