BERT

The original famous encoder that learns by filling in blanked-out words.

BERT (Bidirectional Encoder Representations from Transformers, 2018) is the model that made the read-both-directions, understand-don't-write recipe famous. It's pretrained on huge text by masked language modeling — hide some words and predict them — which forces it to build deep contextual embeddings. Once pretrained, BERT is fine-tuned for specific jobs (sentiment, question answering, tagging) by adding a tiny head on top, the same transfer-learning idea as Book 01's fine-tuning, just on an encoder instead of a writer.

Appears in

Two ways to read Masked Language Models · pt 1
Two sentences at once Masked Language Models · pt 4

Nearby in the brain

attention bidirectional causal mask encoder [CLS] token dot product natural language inference next sentence prediction