Masked Language Models · Part 4 / 5

Two sentences at once

Single words are not enough. Does sentence B follow sentence A? Is it a paraphrase? For that, the model reads pairs.

Filling in a blanked-out word is about one sentence. But lots of questions live between two of them: does B follow A? Is B a paraphrase of A? Does A imply B? (That last one has a name — entailment.)

Reading a pair

So feed the model two sentences at once. It needs to know where one ends and the other begins, and which token belongs to which. BERT does this with two cheap tricks.

• A [SEP] token sits between them (and at the very end) — a literal full-stop the model can see.
• A segment embedding tags every token as segment 0 (sentence A) or segment 1 (sentence B), then gets added on — exactly like the position vectors back in Book 01.

And one special token leads the whole input: [CLS]. Its final vector is trained to be a summary of everything — read both sentences, squeeze the verdict into that one slot.

Does B follow A?

Next Sentence Prediction: half the time B really is the next sentence; half the time it’s a random one. The model reads the [CLS] vector and calls it — IsNext or NotNext.

Try it: swap in the random B. The segment tags don’t change — but the [CLS] summary flips to NotNext. The whole verdict rides on that one front-of-the-line vector.

Next: stop predicting words, start labelling them.