principles.fyi · the brain · concept

induction head

When the model sees a pattern start again, it predicts what came next last time.

...A B...A -> B

An induction head is a little machine inside the model, built from its attention, that finishes patterns. When it sees a word it has seen before, it peeks back at that earlier spot, notices what word came right after it, and guesses the same word will come next now. This is much of how a model "learns" from examples you give it in the prompt, without any extra training.