principles.fyi · the brain · concept

[MASK] token

A special placeholder word that stands in for a hidden word the model must guess.

"the [MASK] sat" -> predict the word at [MASK]

The [MASK] token is a single special entry in the vocabulary that replaces a word during masked language modeling, marking the slot the model has to fill in. There's a subtlety: if the model only ever saw [MASK], it would behave oddly at real-use time when no [MASK] is present — so of the 15% of tokens BERT picks to predict, it replaces 80% with [MASK], swaps 10% for a random word, and leaves 10% unchanged. That mix keeps the model honest about every position, not just the obviously-masked ones.