Masked Language Models · Part 5 / 5

Put it to work

A pretrained encoder plus a small head, fine-tuned on labels, does the jobs the Book 01 writer never could: classify, compare, and tag.

The already reads — it grew rich for free, just by un-masking text.

Now we cash that in.

Borrow the brain, add a hand

Keep the whole pretrained encoder. Bolt one tiny new layer on top — a head — and train it on a small pile of labeled examples. That’s : the reading is already done, the head only learns to point.

pretrain on raw text (free, huge) → add a small head → on labels (cheap, small)

Three shapes of job

one piece of text — is this review happy or sad? Read the one summary vector at the slot, hand it to the head, out comes a label.

Compare two pieces of text — : does sentence A entail, contradict, or stay neutral toward B? Same trick — now [CLS] holds the relationship.

every word — find the people, places, and companies. Here the head reads each token’s vector, not just one.

Naming spans needs a code for where a thing starts and stops. That’s : B begins an entity, I is inside it, O is outside — plus the kind (person, place, org).

Try it: pick the Tim Cook sentence and hover his name — “Tim” goes B, “Cook” goes I, and the run lights up as one entity.

Go deeper: why one head per shape

Classification reads one vector (the [CLS] summary) and emits one label. Tagging reads every token vector and emits one label each — same encoder underneath, only the head’s shape and what it points at change. Fine-tuning nudges the whole stack a little so the read sharpens for your task.

The two books

Book 01 read left-to-right to write the next word. Book 02 read both ways to understand — and now to classify, compare, and label.

The decoder talks. The encoder gets it.

Next: what they’re made of

We’ve built two machines. Book 03 opens them up — what these models are actually made of, and the strange, lawful ways they behave at scale.

Back to the contents →

Sources · 4