Masked Language Models

BERT-style encoders use context on both sides of a token. Explore masked-token training, contextual representations, and tagging.

Two ways to read — Book 01 read left-to-right to write the next word. There is another way — read both directions at once, to understand.
Fill in the blank — With both directions visible, next-word guessing is trivial. So we hide a word in the middle and guess that instead.
Meaning, in context — Book 01 gave each word one fixed arrow. But “mouse” the animal and “mouse” the gadget should not share a point.
Two sentences at once — Single words are not enough. Does sentence B follow sentence A? Is it a paraphrase? For that, the model reads pairs.
Put it to work — Add a task-specific output layer to a pretrained encoder, then fine-tune it on labeled examples for classification, sentence comparison, or token tagging.

Sources and further reading are in the bibliography.

Contents