principles.fyi · the brain · concept

Common Crawl

A free, enormous snapshot of the public web that most LLM training data starts from.

raw web pages (HTML) -> filter & clean -> training-ready text

Common Crawl is a nonprofit that has been crawling and publishing the open web since 2008 — petabytes of raw HTML pages. It's the single biggest raw source feeding LLM pretraining, but it is messy: spam, boilerplate, navigation junk, duplicates, and toxic content all mixed together. Almost nobody trains on it directly; instead teams aggressively filter and clean it. Cleaned-up datasets like C4 and Dolma are essentially 'Common Crawl, scrubbed'.