250+ billion web pages (petabytes) — the raw material behind most LLM pretraining.
Browse commercial NLP / Text → Visit original source ↗Common Crawl is a nonprofit that publishes monthly web crawls. The cumulative archive holds 250B+ web pages across multiple petabytes of WARC files. Virtually every large language model — GPT, LLaMA, PaLM — is pretrained on some filtered/cleaned subset of Common Crawl (C4, RefinedWeb, FineWeb, etc.).
LQS is our 7-dimension quality score, computed from the dataset's published statistics. See methodology →
Composite score computed from the 7 dimensions below: completeness, uniqueness, validation health, size adequacy, format compliance, label density, and class balance.
Common tasks and benchmarks where Common Crawl is the default or competitive choice.
What's actually in the dataset — from the maintainer's published stats.
Common Crawl is distributed under Terms of use (redistribute with attribution). This is a third-party public dataset; LabelSets indexes and scores it but does not host or redistribute the data. Always verify current license terms with the maintainer before commercial use.
LabelSets sellers offer paid nlp / text datasets with what public datasets often can't give you:
Other entries in the NLP / Text catalog.