Transformers, ELI5 · Part 2 / 10

Meaning is a direction

Turn each word into a vector and meaning becomes geometry — similar words point the same way, and scoring a word is just a dot product.

A computer multiplies numbers, not words. The fix is the key idea in the whole field.

A word becomes a list of numbers

Give every word its own list of numbers — a , its (4 here; thousands in a real model).

cat → [0.9, 0.3, -0.2, 0.6]
dog → [0.8, 0.4, -0.1, 0.5]

Nobody assigns these by hand — the model learns them.

see this in PyTorch

Each token id is just a number that looks up its own learned vector in a big table, turning a row of ids into a stack of meaning-vectors.

import torch
import torch.nn as nn

vocab_size = 128000   # how many distinct tokens exist (the whole "dictionary")
d_model = 64         # how big each token's vector is (its "meaning" in numbers)

# The learned lookup table: one row per token, each row is a d_model vector.
# embedding.weight has shape (vocab_size, d_model) and is trained like any layer.
embedding = nn.Embedding(vocab_size, d_model)

# Our input is just token ids — integers pointing at rows in the table.
ids = torch.tensor([42, 7, 1001])   # ids: (seq,)  here seq = 3

# Look up each id -> grab its row -> stack into one tensor.
vectors = embedding(ids)   # vectors: (seq, d_model) = (3, 64)

# So token id 42 became a 64-number vector, id 7 its own vector, etc.
print(ids.shape)      # torch.Size([3])
print(vectors.shape)  # torch.Size([3, 64])

Go deeper: how does text even become numbers?

The model never sees letters. Text is first chopped into (whole words, or fragments like straw+berry), each a row-number in a fixed dictionary — and that number is what gets looked up.

Go deeper: how do the numbers learn their meaning?

They start as random noise. Next-word nudges them — like any weight — until words used alike drift together. Meaning is a side-effect of getting predictions less wrong (nobody writes it down). word2vec did this as a separate step years ago; modern models fold it into the one big model.

A list of numbers is an arrow

A vector is a point in space: the tip of an arrow. So meaning becomes geometry:

Similar words → arrows pointing the same way (cat ≈ dog).
Direction carries meaning. Length is how loudly a word votes.

Scoring a guess is a dot product

Does word w fit here? Take the context arrow h and the word arrow w, multiply position-by-position, add it up: the . Bigger = better fit.

Try it: drag toward 10–15°. The dot product and pick different winners — the longer arrow wins. Length is a real vote.

see this in PyTorch

A matrix multiply (@) takes a grid of numbers shaped (n, d) and a grid shaped (d, k) and produces a new (n, k) grid, where the inner sizes must match and each output is a row dotted with a column.

import torch

# A tensor is just a grid of numbers (like a spreadsheet of values the model learns from).
A = torch.randn(2, 3)   # A: (n, d) = (2 rows, 3 cols) -> 2 tokens, each a 3-number vector
B = torch.randn(3, 4)   # B: (d, k) = (3, 4)            -> turns each 3-vector into a 4-vector

# The shape rule for @ : (n, d) @ (d, k) -> (n, k)
# The inner d's MUST match (3 == 3); they "cancel", leaving the outer (n, k).
C = A @ B               # same as torch.matmul(A, B)
print(C.shape)          # torch.Size([2, 4])  -> 2 tokens, now described by 4 numbers each

# Each output number is a row of A "dotted" with a column of B:
# C[0, 0] = A[0, 0]*B[0, 0] + A[0, 1]*B[1, 0] + A[0, 2]*B[2, 0]  (multiply pairs, then add)
print(torch.allclose(C[0, 0], (A[0] * B[:, 0]).sum()))  # True

Where each word sits

A word’s vector is the same wherever it appears — so “dog bites man” and “man bites dog” look identical. Fix: give each a vector too, and just add it on.

Try it: flip the sentence. The word rows reorder, the position rows don’t — so the input matrix differs. That’s the only thing telling the two sentences apart.

Next: how words borrow meaning from their neighbours.

Sources · 6

Full bibliography →