principles.fyi · the brain · concept

quantization

Saving the model's numbers more roughly, so it fits in less space.

32 bits per number -> 8 bits per number = ~4x smaller

A model is made of millions of numbers (its weights), and normally each one is stored very precisely, using lots of tiny on-off switches. Quantization rounds those numbers to fewer switches, like writing "3.1" instead of "3.14159265." The model gets much smaller and faster to run, and because there are so many numbers helping each other, rounding them a little barely changes the answers.