Transformers, ELI5 · Part 6 / 10

Turning a vector back into words

One last matrix multiply scores every word in the vocabulary — the Chapter 2 dot product, run against the whole dictionary at once.

After the last block, the model holds one vector — the final word’s hidden state — call it h. Still just numbers. Now turn it into a vote over words.

Our tiny dictionary

Sentence so far: “the cat sat”. Real models pick from a vocabulary of tens to hundreds of thousands of tokens; we’ll pretend the dictionary is just six — all ways to finish the sentence:

on · down · there · still · up · quietly

Score each, pick the best. The only input is h.

One dot product per word

From Chapter 2: score a word by dotting h with that word’s arrow. Do it for every word at once — multiply h by the unembedding matrix Wᵤ (each column is one word). Out come logits: one raw score per word.

Try it: the bottom row is the six scores. Click one — it lights up h and that word’s column and writes out the dot product. Bigger = the model likes that word more.

import torch

# h: the final vector for the LAST token        # h: (d_model,)
# E.weight: the embedding table, shape (vocab, d_model)
# here we TIE weights (reuse the input table); many large models learn a separate output table instead
logits = h @ E.weight.T      # (d_model,) @ (d_model, vocab) -> (vocab,)
# one raw score per word in the vocabulary; biggest = top guess
next_word_id = logits.argmax()

Embedding: word → vector, going in. Unembedding: vector → words, coming out.

Raw scores aren’t probabilities yet. Last step, next.