principles.fyi · the brain · concept

grouped-query attention (GQA)

Let many query heads share a smaller set of keys and values, to shrink the KV cache.

64 query heads share 8 key/value groups (Llama 3 70B); 1 shared group = MQA

Each attention head normally keeps its own keys and values, and the KV cache has to store all of them — which gets expensive for long text. Grouped-query attention keeps the query heads separate (so the model still asks many different questions) but lets groups of them SHARE one set of keys and values, so there's far less to cache. Llama 3 70B, for instance, has 64 query heads but only 8 key/value groups. Sharing a single set across every head is the extreme version, multi-query attention (MQA); DeepSeek's MLA compresses the cache a different way. Same idea as plain multi-head — just much cheaper to run.