Two ways to read

In Book 01 the model read left-to-right and wrote the next word. It hid the future with the causal mask — peeking ahead would be cheating.

That model is a writer. This book builds its sibling: a reader.

The fork

A writer reads one direction, because it only ever needs the past.

Bidirectional attention lets an earlier position use later context. For example, the representation at “not” can incorporate the later word “boring.” A causal decoder can already use the earlier “not” while processing “boring.”

So make one change: drop the mask. Now attention looks left and right — it reads both directions. That model is an encoder. The famous one is BERT.

Same sentence, two readers. The decoder's causal mask hides the future; the encoder removes it, so every word sees the whole sentence. The grid is the mask itself — the only thing that changed.

Try it

Focus on “not”, then switch to Encoder. Its representation can now use the later “boring.” At the “boring” position, both arrangements can use the earlier “not.”

This diagram isolates the attention mask. Actual encoder and decoder models can also differ in training objectives, position representations, normalization, and output layers.

Today

Decoder LLMs dominate text generation today. But encoders quietly power search, embeddings, and classifiers under the hood. BERT (2018) kicked this off, and refreshed recipes (e.g. ModernBERT, 2024) keep encoders fast and useful.

So what do we train it on?

A writer trains on “guess the next word.” But if our reader can see the whole sentence at once, that game is trivial — the answer is right there.

We need a different game. Next.

Sources · 5

1. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). arXiv:1810.04805.
2. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models).
3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NeurIPS 2017). arXiv:1706.03762.
4. Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Cooper, N., Adams, G., Howard, J., & Poli, I. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663.
5. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP 2019. arXiv:1908.10084.

Full bibliography →