principles.fyi · the brain · concept

positional encoding

Extra signals added to each word's embedding so the model knows their order.

input = word_embedding + position_vector

A transformer reads all words at once, so on its own it can't tell "dog bites man" from "man bites dog." Positional encoding fixes this: each position gets its own fixed vector of numbers, which is added to that word's embedding (same size, added element-wise) before processing. The original transformer built these vectors from sine and cosine waves of different wavelengths, so nearby positions get similar patterns and the model can learn to compare positions — including their relative distances. The result is one combined vector per word that carries both what the word means and where it sits. Most modern models skip this add-on and use RoPE (rotary position embedding) instead — they rotate each word's query and key by an angle set by its position, so attention sees only how far apart two words are (a few use ALiBi). That relative view is what lets them stretch to very long text.