principles.fyi · the brain · concept

multi-head attention

Several "lookers" read the sentence at once, each chasing a different kind of clue.

MultiHead(Q,K,V) = Concat(head_1, ..., head_h) W_o

Multi-head attention runs several attention operations side by side, each with its own learned projections for deciding which words to pull information from. The point is that one relationship isn't enough: one head might track "who does the verb," another "which noun this adjective describes," another long-range topic links (these are illustrative — real heads don't always split up this cleanly). Each head produces its own slice of output; the slices are concatenated and then mixed by a learned matrix W_o into the final result. So a layer can attend to many relationships at the same time instead of being forced to pick one.