principles.fyi · the brain · concept

masked language modeling

Hide some words in a sentence and train the model to guess them from both sides.

loss = -log P(true_word | sentence with that word masked), averaged over masked positions

Masked language modeling (MLM) is the training game that powers encoders: randomly hide a fraction of the tokens (BERT hides about 15%) and ask the model to predict the originals using the surrounding context on both sides. It's the bidirectional cousin of Book 01's next-word prediction — but because the model can peek left AND right, it can't cheat by copying the next word, so the task pushes it to truly understand each blank. The loss is the same cross-entropy used in Book 01, just measured only on the masked positions.