principles.fyi · the brain · concept

mixture of experts (MoE)

A model whose layers each keep many helper networks, but route every word to just a few.

active compute per word ≈ (experts used, e.g. top-2) / (experts per layer, e.g. 8), summed over every MoE layer — total params big, params used per word small.

A normal layer runs every word through one feed-forward network. MoE swaps that for many parallel "expert" networks plus a tiny router. The router scores the experts for each word and sends it to just the top one or two; the others sit idle for that word. This happens independently at each MoE layer, so a single word can hit different experts as it moves up the stack (attention and the rest of the model stay fully active). The payoff: total parameters can be enormous, but the compute per word stays close to a small dense model, because only a fraction of the weights fire for any given word.