“When Mary and John went to the store, John gave a drink to ___.” You know the answer is Mary. So does GPT-2 small, a 124-million-parameter language model. This page follows that one sentence through the model, step by step.
Nothing here is a mock-up. Every grid, bar and number is what GPT-2 actually computed on this sentence.
Scroll to play each step. ← → also step through.
Want the same steps with the math written out and small numbers you can check by hand? Read Transformers, ELI5.
Model: GPT-2 small (OpenAI, 2019), run in full precision. Layer and head numbers count from 0, as in the interpretability literature. The 2-D word map is a principal-component projection of the token rows.