principles.fyi · the brain · concept

anisotropy

When a model's vectors all crowd into a narrow cone instead of spreading out evenly.

vectors cluster in a narrow cone -> cosine-similarity inflated even for unrelated words

Anisotropy describes contextual embeddings that, rather than filling the space in all directions, pile up in a thin cone — so even unrelated words end up with surprisingly high cosine similarity. This is a wrinkle Book 01 didn't have to worry about much: it means raw cosine-similarity between two contextual vectors can be misleading, because almost everything looks somewhat alike. People correct for it with tricks like subtracting the mean vector or whitening before comparing.