Two sentences at once

Single words are not enough. Does sentence B follow sentence A? Is it a paraphrase? For that, the model reads pairs.

Filling in a blanked-out word is about one sentence. But lots of questions live between two of them: does B follow A? Is B a paraphrase of A? Does A imply B? (That last one has a name — entailment.)

Reading a pair

So feed the model two sentences at once. It needs to know where one ends and the other begins, and which token belongs to which. BERT does this with two cheap tricks.

• A [SEP] token sits between them (and at the very end) — a literal full-stop the model can see.
• A segment embedding tags every token as segment 0 (sentence A) or segment 1 (sentence B), then gets added on — exactly like the position vectors back in Book 01.

And one special token leads the whole input: [CLS]. Its final vector is trained to be a summary of everything — read both sentences, squeeze the verdict into that one slot.

Does B follow A?

Next Sentence Prediction: half the time B really is the next sentence; half the time it’s a random one. The model reads the [CLS] vector and calls it — IsNext or NotNext.

Pick sentence B. Watch [SEP] split the pair, the segment 0 / segment 1 tags get added on, and the [CLS] vector light up to vote IsNext or NotNext.

Try it

swap in the random B. The segment tags don’t change — but the [CLS] summary flips to NotNext. The whole verdict rides on that one front-of-the-line vector.

Today

NSP turned out to be too easy to be worth much — RoBERTa dropped it and got better. To compare sentences for real (search, “find me similar questions”), today we use sentence-embedding models like SBERT: encode each sentence to one vector, then just dot-product them — the Book 01 trick, at the sentence level.

Next: stop predicting words, start labelling them.

Sources · 4

1. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). arXiv:1810.04805.
2. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692.
3. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP 2019. arXiv:1908.10084.
4. Williams, A., Nangia, N., & Bowman, S. R. (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL-HLT 2018 (pp. 1112–1122). arXiv:1704.05426.

Full bibliography →