Fill in the blank

Book 01’s model read left to right and guessed the next word. It had to — the rest of the sentence was its own future, hidden behind the causal mask.

This model sees the whole sentence at once, both directions. So the next-word game breaks.

Now next-word is cheating

the cat sat on the ___

In next-token training, each position predicts the following token. If attention can read that following position in the original sentence, it can access the target directly. Hiding the target prevents this shortcut.

New game: hide a word in the middle

Cover a word inside the sentence and ask the model to recover it, using everything on the left and the right.

the cat ___ on the mat → sat

This is the prediction task. It has an old name — the cloze task, a fill-in-the-blank test — and a model trained this way is doing masked language modeling.

Think of it as repairing a damaged sentence: we add the damage on purpose, the model learns to undo it. This is called denoising.

A sentence with one word blanked out. The model reads both sides and ranks its guesses for the hole — the true word lights up in amber.

Try it

drag the blank to different words. A word boxed in by neighbours (cat ___ on) is easy; a wide-open one is a real guess. Same as Book 01’s training — just with the hole in the middle instead of the end.

Same scoring as before

Grading works exactly like Book 01: the model gives every word in the vocabulary a probability, and cross-entropy punishes it for putting low probability on the true word. The only twist — we score only the hidden positions. The untouched words are just there for context.

Go deeper: how BERT actually picks the holes

BERT selects 15% of input tokens as prediction targets.

Replacing every target with [MASK] creates a mismatch: ordinary text used for downstream tasks does not contain that token.

So of the ~15% picked:

• 80% → replaced with [MASK] — the main fill-in-the-blank drill
• 10% → replaced with a random word — now it must double-check words that look fine
• 10% → left unchanged — so it keeps a real opinion at every position, flag or no flag

The BERT masking procedure includes unchanged and randomly replaced targets to reduce this mismatch. The model predicts the original token at every selected position.

Today

This bidirectional fill-in-the-blank powers the understanding side of NLP — search, classification, tagging. The text-writing models you chat with kept Book 01’s left-to-right next-word game. Same transformer parts, two different training games.

Next: why reading both directions gives a word a sharper meaning than reading one.

Sources · 4

1. Taylor, W. L. (1953). “Cloze Procedure”: A New Tool for Measuring Readability. Journalism Quarterly, 30(4), 415–433.
2. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). arXiv:1810.04805.
3. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models).
4. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI technical report.

Full bibliography →