principles.fyi · the brain · concept

contextual embedding

A word's vector that changes depending on the sentence it's sitting in.

contextual_embed(word) = encoder's hidden state for that word in this sentence

A contextual embedding is the vector an encoder produces for a token AFTER attention has mixed in its neighbors — so the same word gets a different vector in different sentences. This is the upgrade over Book 01's plain embedding (one fixed vector per token from a lookup table): here 'bank' near 'river' lands in a different spot than 'bank' near 'money'. It's really just the token's hidden state at the top of the encoder, and it's what makes encoders good at meaning-sensitive tasks.