principles.fyi · the brain · concept

scaling laws

Bigger models keep getting better in a steady, predictable way.

loss ≈ a / (size ^ b) (b is a small positive number)

A model's "loss" is just how wrong its guesses are, on average. Scaling laws are the surprising discovery that this wrongness shrinks along a smooth, ruler-straight path when you draw it on a stretched chart, as you add more parameters, more data, or more compute. Because the line is so straight, you can extend it and guess how good a much bigger model will be before you spend the time and money to build it. That lets people plan giant models on purpose instead of by luck.