principles.fyi · the brain · concept

GeLU

A smooth on/off switch for neurons that gently fades small values toward zero instead of cutting them off sharply.

GeLU(x) = x * Phi(x), where Phi(x) = P(X <= x) for X ~ Normal(0,1)

GeLU (Gaussian Error Linear Unit) is a common nonlinearity used inside a transformer's feed-forward layer, deciding how much of each value to pass forward (some models instead use ReLU or gated variants like SwiGLU). Instead of ReLU's hard rule (negatives become 0, positives pass unchanged), it multiplies each input by a smooth "keep probability" that rises from 0 to 1 as the input grows: GeLU(x) = x * P(X <= x), where X is a standard bell curve (the normal distribution). That smoothness means small negatives leak through a little and the curve has no sharp kink, which gives cleaner gradients and tends to train slightly better. The feed-forward layer is where the network does much of its "thinking" between attention steps, reshaping each token's features.