principles.fyi · the brain · concept

gated FFN (SwiGLU)

Two lanes meet: one scales the other up, down, or off.

out = (x·W1 * swish(x·W3)) · W2

The feed-forward step makes two copies of the same input with two different linear lanes. One lane is the signal; the other passes through swish to become a learned gain that multiplies the signal point by point. Unlike a 0-to-1 dimmer, the swish gate is mostly positive but unbounded above (so it can amplify, not just attenuate) and dips slightly below zero, so each number can be turned up, down, off, or gently flipped. So instead of ReLU's hard rule ("keep it or zap it to zero"), the amount is smooth and learned. That richer, learnable control lets the network pass more expressive signals; the hidden width is trimmed (about two-thirds) to keep the parameter and compute budget comparable, and it still wins, which is why modern Transformers (e.g. LLaMA, PaLM) use it.