principles.fyi · the brain · concept

vector

A list of numbers that pins down a spot in space.

v = [0.2, -1.4, 0.7]

A vector is just an ordered list of numbers, like [0.2, -1.4, 0.7], which you can picture as a point (or an arrow from the origin) in space — three numbers give a spot in 3D, and a transformer uses hundreds or even thousands of numbers for a point in that-many-dimensions space. This works because position carries meaning: words with similar meanings land near each other, so "math" can ask "distance" and "similarity" instead of comparing letters. In a transformer every token (a word or word-piece) becomes a vector, and the whole model's job is to nudge these vectors around, layer by layer, until each one captures what the token means in its context.