Looking inside a model

Can we see what a trained model actually does inside? A little — the field is called interpretability.

Go deeper: why reading the model is hard (superposition)

A neuron can respond to several patterns, and a pattern can be represented across several neurons. Superposition studies overlapping representations. A sparse autoencoder can find useful candidate features, but does not guarantee a complete or uniquely correct dictionary of a model’s concepts.

Cramped overlapping features; toggle 'untangle' to separate them.

Induction heads: copy the pattern

One real, discovered circuit — an induction head: spot a repeat, then copy what came after it last time. …A B … A → B.

Step through: find the earlier match, copy what followed it.

The logit lens: watch the guess form

The logit lens: apply the unembedding at every layer, not just the last, to inspect candidate token scores at intermediate stages. The scores need not improve monotonically or end at a correct answer.

This chosen example shows a prediction changing across layers. It does not establish that every model’s predictions become steadily more correct.

What the logit-lens calculation needs

Take the hidden state at the position and layer being inspected. Apply the model’s appropriate final normalization and output projection, then compare the token scores. Repeating this for intermediate states creates a logit-lens view. The output projection was trained for final states, so reading earlier states through it has limitations.

What this lets you conclude

guess the next word → vectors → attention → the residual stream → stacking → training → scale → a peek inside

A plausible interpretation is a hypothesis about a computation. Stronger evidence comes from changing or removing the proposed component and measuring the effect. A colorful attention map or a confident final token alone does not establish the explanation.

Return to the Transformer overview or read about evaluation and failure.

Sources · 5

1. Olsson, Catherine, et al. “In-context Learning and Induction Heads.” Transformer Circuits Thread, 2022.
2. nostalgebraist. “interpreting GPT: the logit lens.” LessWrong, 2020.
3. Elhage, Nelson, et al. “Toy Models of Superposition.” Transformer Circuits Thread, 2022.
4. Bricken, Trenton, et al. “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning.” Transformer Circuits Thread, 2023.
5. Jurafsky, Daniel, and James H. Martin. Speech and Language Processing (3rd ed. draft, 2026), Chapter 8: “Transformers.”

Full bibliography →