One space for words and pictures

An image encoder and a text encoder produce vectors. Training them together can make a matching image and description receive a high similarity score.

CLIP used 400 million collected image–text pairs. The accompanying text supplies supervision without requiring a new category label for every image. The pairs can still be noisy or inaccurate.

Prefer the collected pair

Encode every image and caption, then score all combinations. In a batch of N pairs, there are N positive pairings and N² − N pairings treated as negatives. The loss encourages a collected pair to score higher relative to the alternatives in its row and column. Shared parameters mean an update need not improve every pair.

This is contrastive learning. Unlike a fixed-category classifier, CLIP can score new text descriptions against images. That flexibility does not guarantee that it interprets every description correctly.

Handing a picture to a language model

Image–text alignment gives a useful vision representation. Its vectors do not automatically belong to an arbitrary language model’s embedding space; connecting the two requires additional modeling and training.

One approach uses a learned projection to map vision features to the language model’s input width, then places those visual tokens alongside text embeddings. Training teaches the combined system to use them. Other architectures use different connectors or cross-attention.

The bill is real: a 336×336 image at 14×14 patches is 576 tokens before the user has typed a word. Hence compression schemes — a small set of learned queries that pull, say, 32 vectors out of the 576 (the Q-Former). Detail traded for room, which is worth it for many images in one conversation and not worth it for reading one dense page.

The loop, pointed at a screen

Chapter one’s agent could only ever handle text. Give it eyes and the set of jobs changes.

That is why browsing agents look at screenshots: rendering is the one format that cannot hide anything. Same for documents — a PDF’s layout, its tables, its diagrams are all visible and all invisible to text extraction.

And it brings a failure mode text never had. Asked whether the page has an Export CSV button, the model says yes. Dashboards usually do. The answer came from what dashboards are normally like rather than from what is in this picture — visual hallucination, measured by POPE, which does exactly this: names an object and asks whether it is present.

For an agent this is worse than a wrong sentence. It clicks empty space, observes nothing, and reports success.

Go deeper: designing for a wrong observation

The mitigations are the same shape as everywhere else in this book — assume the observation may be false and build around it.

Make it describe before it acts. Naming what is on screen first is much harder to fake than answering yes to a leading question.

Check the click did something. The screen after should differ from the screen before. If it does not, the button was not there.

Never let I clicked it become it worked. That substitution is the actual harm; the hallucinated button is only how it started.

The model can now read a picture. It still cannot make one — nothing so far produces pixels.

Sources · 4

Full bibliography →