principles.fyi · the brain · concept

attention

Each word looks at all the words (itself included) and pulls in what's relevant.

weights = softmax(query · key); output = sum(weights * values)

Attention is how a token builds a custom summary of all the tokens (itself included), blended by relevance. Each token sends out a query ("what am I looking for?") and every token offers a key ("what I'm about"); matching a query to a key gives a score for each pair, those scores are turned by softmax into weights that add up to 1, and each token then collects a weighted mix of every token's value vector. It works because relevance gets decided on the fly from the actual words present, so "it" can reach back and grab "the dog" instead of "the bone" (and a token can also keep a lot of its own content by weighting itself). This is the only step where tokens exchange information; everything else processes each token alone, so attention is where context actually enters.