softmax

Turns a list of scores into probabilities that add up to 1.

softmax(x)_i = e^(x_i) / sum_j e^(x_j)

Softmax takes any list of real numbers and turns them into positive values that sum to 1, so they read as probabilities. It does this by exponentiating each score and dividing by the total: softmax(x)_i = e^(x_i) / sum_j e^(x_j). Because exponentials grow fast, a slightly bigger score gets a much bigger share — it amplifies the winner while still giving the losers a small voice. In a transformer, attention uses this to turn raw match scores into weights that decide how much each word should listen to every other word.

Appears in

Stack it ~96 times Transformers, ELI5 · pt 5
From scores to a word — and the loop Transformers, ELI5 · pt 7
Evidence adds points The Math Beneath · pt 3

Nearby in the brain

temperature Bradley-Terry model cross-entropy loss odds reward sigmoid layer norm matrix multiply