principles.fyi · the brain · concept

RMSNorm

Like layer norm, but skips the centering step — just divide each vector by its overall size.

y = x / sqrt(mean(x^2) + eps) * gain

RMSNorm is a lighter cousin of layer norm. Where layer norm first subtracts the mean (re-centers to 0) then divides by the spread, RMSNorm drops the centering (and the bias) entirely: it just divides the vector by its root-mean-square — a measure of its overall size — and applies a learned per-feature gain. It turns out the re-centering wasn't pulling much weight, so dropping it is cheaper and trains just as well. It's the default in most modern open models (Llama, Mistral, Qwen, Gemma, DeepSeek).