Transformers, ELI5 · Part 1 / 10
The one thing a language model does
A language model does one thing — guess the next word, then do it again. Everything else is machinery for that one guess.
Forget “AI.” A language model has one job:
Read some text. Guess the next word.
Give it “the cat sat” and it guesses on. Then it sticks on on the end and guesses again.
The loop that writes everything
That loop is — every sentence a model writes, run fast.
So “understand a transformer” really means: understand that one guess.
What’s inside the guess
Almost the whole model is one operation: . Its atom is the — multiply two lists of numbers pairwise, add them up, get one number. A matrix multiply is just a whole grid of those, done billions of times.
We mark every matrix multiply in amber, so you can see how much of the machine is really just that.
First, one problem
A matrix multiply needs numbers. Our input is words. A computer can’t multiply cat.
So step one: turn a word into numbers that capture its meaning. Next.