What they eat

A model is mostly what it read. So what did it read — and who decided what stays in?

A model has no facts of its own. Everything it “knows” is a faint echo of text it read once.

That reading is — the giant first training run from Book 01, here seen from the other side: not the loss, but the diet.

So what’s on the plate?

Mostly the web. The raw fire-hose is — petabytes of public pages, scraped and dumped as-is. Huge, and a mess: brilliant writing sitting next to spam, cookie banners, and worse.

Nobody trains on it raw. They clean it first.

Try it: poke the biggest slice, then hit the filter. Boilerplate, near-duplicates, and toxic text drop out — and the kept pile is now denser, not just smaller.

Cleaning is two jobs

Quality — drop the boilerplate, the spam, the same page scraped a thousand times. Less data, but every byte is worth learning from. Cleaning doesn’t just shrink the diet — a denser pile makes the model better.

Safety — drop toxic text and personal info before the model ever reads it. A small slice by size (the amber one), but the slice you most want gone.

Curated mixes go further than plain web text. deliberately blends in books, code, and papers — a richer diet teaches a broader model. (Sets like C4 take the other tack: just very clean web. Newer mixes like Dolma do both.)

The open-data commons is shrinking. As models got valuable, more sites started blocking the crawlers (via robots.txt and paywalls), so the easy, free web that fed early models is quietly closing. Tomorrow’s increasingly run on licensed, bought, or model-generated data instead.

Go deeper: the honest, hard part

Filtering is also where the uncomfortable questions live — and they’re not solved, just argued about:

Copyright. A lot of the web is someone’s work. Is training on it fair use, or copying? Courts are still deciding.
Consent. Most authors never agreed to be training data. robots.txt lets a site opt out — but only going forward, and only if the crawler honors it.
Privacy. Personal info leaks into the scrape. Safety filters catch a lot — never all.
Skew. The web over-represents some languages, regions, and viewpoints. A model trained on it inherits that lopsidedness as its sense of “normal.”

Every one of these is a choice baked into the data — and someone chose. Because so much rides on one pretrained base, those choices ripple out to every app built on top.

A model isn’t a neutral mirror of the world. It’s a mirror of what someone decided to feed it.

That’s also why grading them is so slippery — if a test question was hiding in the training data, the score means nothing. That’s , and it’s the next chapter.

Sources · 6

Full bibliography →