Transformers, ELI5 · Part 5 / 10

Stack it ~96 times

A block hands back the same shape it got — so you stack it. Zoom out, and the whole model is mostly matrix multiplies adding up.

One block hands back the same shape it got. So its output is a legal input to the next block. So you just… do it again.

The residual stream

Picture one wide highway of numbers, start to finish. A block doesn’t replace it — it reads the highway, computes an update, and adds it back on. That running total is the residual stream. (That’s why the width never changes.)

Does it just keep growing? No — the nudges go both ways, so the vector steers instead of ballooning.

Layer norm keeps it clean

And before each block reads the stream, layer norm rescales it — subtract the mean, divide by the spread — so the numbers can’t drift too big or too small.

The whole model, at a glance

Almost every step is a matrix multiply (amber). Softmax, ReLU, and norms are just the thin glue between them.

Now watch the real numbers

Same model, made concrete: a tiny real transformer where every number is computed. Each step shows before → after.

Press Play. The grid changes at every step, predicts a word, then appends it and runs again — the loop that writes text.

Next: how that last vector turns into a word.