segment embedding

An extra tag added to each word saying whether it belongs to text A or text B.

input = word_embedding + position_embedding + segment_embedding

Segment embeddings are learned vectors added to every token to mark which of two inputs it came from — segment A or segment B — so the encoder can keep two texts straight when they're fed in together past a [SEP]. They're added the same way as positional encodings from Book 01 (element-wise onto each token's vector), just carrying 'which chunk' instead of 'which spot'. So a token's input vector ends up being its word meaning + its position + its segment.

Appears in

Two sentences at once Masked Language Models · pt 4

Nearby in the brain

BERT [CLS] token dot product natural language inference next sentence prediction positional encoding [SEP] token