Training data and what gets filtered

Training adjusts parameters using examples. A model can learn reusable patterns and sometimes memorize passages; its output is not simply a lookup of a sentence it read before. What the training examples contain still has a large effect on what it learns.

A dataset is a collection of choices

Training mixtures can include web text, books, code, papers, licensed material, and generated examples. The mixture differs by model and may not be fully disclosed. Common Crawl is one source of crawled web data, not a complete description of every model’s training set.

The sizes and filtering effects illustrate a possible mixture. They are not an audited account of an unspecified model’s data.

What cleaning tries to accomplish

Remove repeated or unwanted material. Deduplication reduces repeated examples. Other filters can remove boilerplate, spam, corrupted text, or documents outside the intended scope.

Reduce harmful or private content. Detection tools may flag personal information or unwanted content. They miss some material and can remove useful examples by mistake; filtering does not guarantee privacy or safety.

Control representation. Choices about language, source, topic, and quality change whose writing is included. A rule based on one writing style can wrongly reject another. More filtering is therefore not automatically better.

Curated datasets such as The Pile publish their source categories. Comparing documented mixtures is more informative than assuming that every language model was trained on the same internet.

Questions that the dataset name does not answer

What permission covers the material? Was personal information retained? Which languages and communities are represented? Did evaluation questions enter training? These questions require evidence about the actual collection and processing steps.

The last issue is data contamination. Overlap can make a supposedly new test easier through prior exposure. Its effect depends on the overlap and evaluation; it does not make every contaminated score meaningless in exactly the same way.

The next page distinguishes the different things an evaluation can measure.

Sources · 6

1. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models).
2. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140), 1–67. arXiv:1910.10683.
3. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., & Leahy, C. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027.
4. Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., et al. (2024). Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In Proceedings of ACL 2024 (Long Papers). arXiv:2402.00159.
5. Longpre, S., Mahari, R., Lee, A., Lund, C., Oderinwale, H., Brannon, W., Saxena, N., et al. (2024). Consent in Crisis: The Rapid Decline of the AI Data Commons. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). arXiv:2407.14933.
6. Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, Ú., Oprea, A., & Raffel, C. (2021). Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633–2650. arXiv:2012.07805.

Full bibliography →