principles.fyi · the brain · concept

denoising

Mess up the text a little, then train the model to repair it.

noisy text -> model -> clean text

Denoising is the general framing behind masked language modeling: corrupt the input (here, by blanking out or swapping some tokens — the 'noise') and train the model to recover the clean original. Learning to undo the damage forces the model to internalize how language normally fits together. The same idea scales up in other models that corrupt whole spans of text and reconstruct them, but plain word-masking is the simplest version.