An image is a sequence too

Strip a transformer back to what it actually requires. A sequence of vectors, and attention letting every position draw on every other.

That is it. No part of the machinery mentions language. Words only got there because we turned them into vectors first.

So an image is not a different kind of problem. It is the same problem with one open question: what should one position be?

One pixel or one patch?

Try one pixel per position. The first problem is the number of positions.

Full attention scores every pair of positions, so that part of its work grows with the square of the token count. One pixel contains only color values; attention can combine pixels into larger patterns, but doing so over 50,176 tokens is expensive. Patches give the transformer a shorter sequence to work with.

Make the unit bigger. Cut the image into squares — 16×16 pixels is one common choice — flatten each into a list of numbers, and multiply by one learned matrix to get a vector of the model’s width. That is a patch embedding, and it is the same lookup-and-project move that turns a word into a vector, with a matrix multiply standing in for the dictionary.

224 ÷ 16 = 14 across and 14 down, so 196 tokens. An ordinary sequence length. Feed it to an unmodified transformer encoder and you have a vision transformer.

Two things the cutting broke

Position is gone. Shuffle the patches and the set is identical, exactly as “dog bites man” and “man bites dog” were identical before positions were added. Same fix: give every grid slot its own learned vector and add it on. Curiously, numbering the patches 1…196 in a straight line works as well as encoding the true 2-D grid — the model works out the shape from the data.

There is no summary. Every patch describes its own square; nothing describes the picture. So append one extra learned vector that starts out meaning nothing, let it attend to everything for twelve layers, and read the classification off it — the CLS token, borrowed wholesale from BERT.

The bill for using a general machine

A convolutional network is built knowing things about images: pixels near each other are related, and a cat is a cat wherever it appears. Those assumptions are wired into the architecture, so they are true before training starts.

A vision transformer assumes none of it. Attention has no notion of nearby; a patch in the corner is, initially, no more related to its neighbour than to one across the image. Every spatial fact has to be learned from data.

Which is precisely why it lost to convolutional networks on ImageNet-scale data and won on JFT-scale — around 300 million images. Fewer built-in assumptions means more to learn and more freedom in what gets learned, and the trade only pays above a threshold. The same bargain language models made: stop hand-designing structure, buy the structure with data.

Go deeper: what changes with patch size?

For a fixed image resolution, halving the patch width and height gives four times as many tokens. Full attention then scores sixteen times as many pairs. Operations applied separately to each token grow fourfold, so sixteen is not the multiplier for every part of the model.

The transformer blocks can keep the same width and number of weights. But an RGB patch contains 3 × p² numbers, so the learned matrix that projects it to the model’s width changes shape when p changes. Learned position embeddings may also need adaptation. Patch size is not generally a runtime setting you can change while keeping every trained weight as it is.

Changing image resolution while keeping patch size fixed is a different operation. Some models support it by interpolating position embeddings onto the new grid. More patches still require more computation.

This gets an image into a transformer. It does not connect it to a single word — the patches and the text still live in unrelated spaces.

Sources · 2

Full bibliography →