next sentence prediction

A yes/no training task: did sentence B actually follow sentence A?

[CLS]-vector -> P(B truly follows A)

Next sentence prediction (NSP) was BERT's second pretraining task alongside masked language modeling: show two sentences and have the model (via the [CLS] vector) decide whether B genuinely followed A in the original text or is a random impostor (the two cases are split 50/50 during training). The goal was to teach relationships between sentences for tasks like question answering. It turned out to add little — later models like RoBERTa dropped it and did better — so it's mostly of historical interest now.

Appears in

Two sentences at once Masked Language Models · pt 4

Nearby in the brain

BERT [CLS] token dot product natural language inference positional encoding segment embedding [SEP] token