Stacking blocks and keeping their shape

Residual additions, normalization, and matrix shapes

One block hands back the same shape it got. That makes its output a valid input to the next block. Different blocks have different learned weights. The number of blocks is a model choice; 96 is an example, not a requirement.

The residual stream

Each sublayer computes an update and adds it to its input. The vectors passed along through those additions form the residual stream. Both terms of each addition must have the same shape.

The number of entries stays fixed. Their magnitude need not: additions may reinforce or cancel. Residual addition alone does not bound the vector’s length.

Normalize what a sublayer reads

In the pre-normalized arrangement shown here, a sublayer reads a normalized version of each token row. Layer normalization subtracts that row’s mean, divides by $\sqrt{\text{variance}+\varepsilon}$ , then applies learned scale and offset parameters. The positive ε prevents division by zero. The residual stream itself still receives the additive update; normalizing its input does not prove the stream stays bounded.

What repeats, and what does not?

A decoder model repeats the attention → add → feed-forward → add block, using different parameters in each block. Token and position lookups happen before the stack; final normalization and vocabulary scoring happen after it.

X_0=\text{token vectors}+\text{position vectors},\qquad X_{\ell+1}=\operatorname{Block}_{\ell}(X_\ell).

Here ℓ labels the block. Every $X_\ell$ has one row per token and the same feature width.

Matrix multiplication performs much of the arithmetic, but its inputs matter:

Operation	What it combines
A projection, such as Q = XW + b	Input-dependent values and fixed learned parameters
QKᵀ	Query and key values that both depend on the input text
AV	Attention weights and value rows, both computed from the input
Normalization, softmax, ReLU, and residual addition	Other operations that change the result; they are not matrix products

The nonlinear operations matter. A complete Transformer is not equivalent to multiplying the original input by one fixed matrix.

Follow the tokens through a complete calculation

The explorer computes one small decoder block with a single attention head. Start with “the cat sat”: three token rows, each with four features. Its fixed, invented weights let us check the calculation; they have not learned language.

Select a token to follow its row through the whole block. Select any number to see the inputs that produced it. For a matrix product, that means one input row multiplied by one weight column. Attention also uses products whose two inputs both depend on the text: Q × Kᵀ computes position-to-position scores, and A × V mixes value rows.

Open the explorer on its own for more room.

Try the causal mask: select the first position, then open “Change the input” and change the final token. The first row’s result stays unchanged because it cannot use future positions. Select the final position to see a row that can use the whole input.

Try the matrix product: select a query entry. Its explanation shows four input values, four entries from one weight column, their products, and the added bias. Now inspect a mixed-value entry: its terms come from different token positions. The arithmetic is similar, but the axes mean different things.

Appending the highest-probability token uses the final position’s output and runs the calculation again with the same weights. Training would be a separate operation that changes those weights.

What if several passes reused the same block parameters? Recurrence explains that variation.

Next: how that last vector turns into a word.

Sources · 7

Full bibliography →