Put it to work

Add an output layer for the task

Keep the pretrained encoder and add an output layer called a head. It maps the encoder’s vectors to outputs such as sentiment labels. Fine-tuning trains the head and can also update the encoder using labeled examples. This is transfer learning: reusing a trained model as the starting point for another task.

pretrain on text → add an output layer → [fine-tune](/glossary#fine-tuning) on labeled examples

Three shapes of job

Classify one piece of text — is this review happy or sad? Read the one summary vector at the [CLS] slot, hand it to the head, out comes a label.

Compare two pieces of text — natural language inference: does sentence A entail, contradict, or stay neutral toward B? Same trick — now [CLS] holds the relationship.

Tag every word — find the people, places, and companies. Here the head reads each token’s vector, not just one.

The widget: tag every word

Naming spans needs a code for where a thing starts and stops. That’s BIO tagging: B begins an entity, I is inside it, O is outside — plus the kind (person, place, org).

Each word gets its own vector, and the head turns each vector into a B / I / O tag. Watch a multi-word name light up as B then I.

Try it

pick the Tim Cook sentence and hover his name — “Tim” goes B, “Cook” goes I, and the run lights up as one entity.

Today

The same recipe scales: one pretrained encoder, swap the head, fine-tune — and you get search, reranking, and the embeddings that feed retrieval. The encoder is the workhorse you rarely see.

Go deeper: why one head per shape

Classification reads one vector (the [CLS] summary) and emits one label. Tagging reads every token vector and emits one label each — same encoder underneath, only the head’s shape and what it points at change. Fine-tuning nudges the whole stack a little so the read sharpens for your task.

The two books

Book 01 read left-to-right to write the next word. Book 02 read both ways to understand — and now to classify, compare, and label.

These training setups give the models different ways to use context. Both can support tasks beyond the ones shown here.

Next: what they’re made of

We’ve built two machines. Book 03 opens them up — what these models are actually made of, and the strange, lawful ways they behave at scale.

Back to the contents →

Sources · 4

1. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). arXiv:1810.04805.
2. Williams, A., Nangia, N., & Bowman, S. R. (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL-HLT 2018 (pp. 1112–1122). arXiv:1704.05426.
3. Ramshaw, L. A., & Marcus, M. P. (1995). Text Chunking using Transformation-Based Learning. In Proceedings of the Third Workshop on Very Large Corpora (pp. 82–94). ACL. arXiv:cmp-lg/9505040.
4. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models).

Full bibliography →