principles.fyi · the brain · concept

KV cache

Save what you already worked out, so each new word is quick.

work per step: no cache ~ n^2, with cache ~ n (n = words so far)

When the model writes the next word, attention compares it against every word so far, using a "key" and "value" it computed for each one. Those numbers never change once a word is fixed, so the model saves them in a little memory box (the cache) and reuses them — it only computes a fresh key and value for the brand-new word. Without the cache, every step would redo the key and value for the whole history again, costing about n-squared work per step; with the cache each step is just n. That's the difference between text crawling out and text flowing.