Transformers, ELI5 · Part 10 / 10

Peeking inside

Can we see what a trained model actually does? A little — induction heads copy patterns, and the logit lens watches a guess form layer by layer.

Can we see what a trained model actually does inside? A little — the field is called interpretability.

Go deeper: why reading the model is hard (superposition)

The catch behind the wins below: a model has more ideas than neurons, so it stacks several into each one — . A can un-tangle them into clean features.

Induction heads: copy the pattern

One real, discovered circuit — an : spot a repeat, then copy what came after it last time. …A B … A → B.

The logit lens: watch the guess form

The : apply the at every layer, not just the last, and watch the guess sharpen from vague to right as it climbs.

see this in PyTorch

Multiply the final vector by the (transposed) embedding table to get one score per word — here we reuse the input table (weight tying); some models learn a separate one.

import torch

# h: the final vector for the LAST token        # h: (d_model,)
# E.weight: the embedding table, shape (vocab, d_model)
# here we TIE weights (reuse the input table); many large models learn a separate output table instead
logits = h @ E.weight.T      # (d_model,) @ (d_model, vocab) -> (vocab,)
# one raw score per word in the vocabulary; biggest = top guess
next_word_id = logits.argmax()

You made it

guess the next word → vectors → attention → the residual stream → stacking → training → scale → a peek inside

Nearly all of it, as promised, is matrix multiplication. That’s the transformer.

Back to the contents →

Sources · 5

Full bibliography →