principles.fyi Part 3 / 5

Masked Language Models · Part 3 / 5

Meaning, in context

Book 01 gave each word one fixed arrow. But "mouse" the animal and "mouse" the gadget should not share a point.

In Book 01, meaning is a direction — and every word got one arrow. One for mouse, forever.

One word, two meanings

But mouse the animal and mouse the gadget are different ideas wearing the same spelling. One fixed arrow has to be both at once — a blurry average. Words with several senses () break the one-arrow idea.

The arrow should depend on the neighbours

A bidirectional reader doesn’t look up a frozen vector. It reads the whole sentence and bakes the context in — a : a fresh arrow for every occurrence.

So the two mousees drift apart. The same spelling lands in different places, and the senses pull into separate clusters — which is exactly what needs.

Each dot is one occurrence of the same word in a different sentence. Static embeddings stack them on a single point; switch to contextual and they split into sense clusters — the cheese-and-tail mice over here, the click-and-cursor mice over there.

Try it: start in static mode — every mouse sits on one dot. Flip to contextual and watch the dot shatter into clusters. Hover a point to read the sentence that put it there, and find the lonely outlier sense.

Go deeper: why the vectors all look alike (anisotropy)

Plot raw contextual embeddings and you get a nasty surprise: almost every arrow points roughly the same way. The cloud isn’t a balanced ball — it’s a thin cone. This is .

The culprit is usually a few rogue dimensions — one or two coordinates that blow up huge on every token and dominate the geometry. They drown out the small, meaningful differences that separate the senses, so everything reads as “similar.”

The fix is gentle: z-score each dimension — subtract its mean, divide by its standard deviation, across the dataset. That deflates the rogue dimensions back to the same scale as the rest, the cone opens into a proper cloud, and the sense clusters you saw above get crisp and far apart.

Same spelling, different point. Next: how a model learns to do this — by playing fill-in-the-blank.

Sources · 5
  1. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models).
  2. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of NAACL-HLT 2018. arXiv:1802.05365.
  3. Coenen, A., Reif, E., Yuan, A., Kim, B., Pearce, A., Viégas, F., & Wattenberg, M. (2019). Visualizing and Measuring the Geometry of BERT. Advances in Neural Information Processing Systems 32 (NeurIPS 2019). arXiv:1906.02715.
  4. Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proceedings of EMNLP-IJCNLP 2019 (pp. 55–65). arXiv:1909.00512.
  5. Timkey, W., & van Schijndel, M. (2021). All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality. In Proceedings of EMNLP 2021 (pp. 4527–4546). arXiv:2109.04404.
Full bibliography →