principles.fyi · the brain · concept

logarithm (log)

A slow-growing curve that counts the digits (the zeros) in a number.

surprise = -log(probability); log(1) = 0; exp and log are inverses, so exp(log(x)) = x

A logarithm asks a backwards question: not "what is 10 used as a factor 3 times?" (that's 1000) but "how many times must I multiply 10 to reach this number?" (that's 3). So it climbs fast at first, then crawls: log(1) is 0, log(10) is 1, log(1000) is 3, and ever bigger inputs add only tiny bits more. Switching the base just rescales the whole curve, so the shape is the same whether you count in 10s, 2s, or e's. We use this for surprise by flipping the sign: surprise = -log(probability). Something certain (probability 1) gives 0 surprise, while a rare event (a tiny probability) sends the log diving down, so minus-log shoots up huge. log and exp are exact inverses, which is why a model's cross-entropy loss is just the minus-log of the probability it gave the true next word.