Product
Help
Pricing
About
Jobs
Log in
Get started
Keyword
Add to collection
Common Crawl
Description:
A nonprofit web-crawling organization that collects and freely provides large-scale snapshots of publicly accessible web pages.
Related topics:
Topic Modeling (17%),
Natural Language Processing Techniques (16%),
Web Data Mining and Analysis (10%)
Wikidata ID:
wiki/Q12055316
OpenAlex ID:
keywords/common-crawl
Language-agnostic BERT Sentence Embedding
2022
·
Fangxiaoyu Feng
, Yinfei Yang
, et al.
·
Proceedings of the conference - Association for Computational Linguistics. Meeting
·
1,921
PDF
Learning Word Vectors for 157 Languages
2018
·
Edouard Grave
, Piotr Bojanowski
, et al.
·
Language Resources and Evaluation Conference (LREC)
·
1,821
PDF
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
2024
·
Guilherme Penedo
, Hynek Kydlícek
, et al.
·
neural information processing systems
·
1,801
PDF
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
2023
·
Penedo, Guilherme
, Malartic, Quentin
, et al.
·
arXiv (Cornell University)
·
1,215
PDF
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
2019
·
Wenzek, Guillaume
, Lachaux, Marie-Anne
, et al.
·
arXiv (Cornell University)
·
960
PDF
DataComp: In search of the next generation of multimodal datasets
2023
·
Samir Yitzhak Gadre
, Gabriel Ilharco
, et al.
·
neural information processing systems
·
908
PDF
Multimodal datasets: misogyny, pornography, and malignant stereotypes
2021
·
Birhane, Abeba
, Prabhu, Vinay Uday
, et al.
·
arXiv (Cornell University)
·
698
PDF
DataComp-LM: In search of the next generation of training sets for language models
2024
·
Jeffrey Li
, Alex Fang
, et al.
·
neural information processing systems
·
628
PDF
Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures
2019
·
Ortiz Suárez, Pedro Javier
, Sagot, Benoît
, et al.
·
Publication Server of the Institute for German Language (Institute for German Language)
·
539
PDF
Large scale distributed neural network training through online distillation
2018
·
Anil, Rohan
, Pereyra, Gabriel
, et al.
·
arXiv (Cornell University)
·
532
PDF
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
2023
·
Hugo Laurençon
, Lucile Saulnier
, et al.
·
neural information processing systems
·
515
PDF
MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
2023
·
Sneha Kudugunta
, Isaac Caswell
, et al.
·
neural information processing systems
·
392
PDF
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
2020
·
Pedro Javier Ortiz Suárez
, Laurent Romary
, et al.
·
Proceedings of the conference - Association for Computational Linguistics. Meeting
·
337
PDF
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
2022
·
Julien Abadji
, Pedro Ortiz Suarez
, et al.
·
Language Resources and Evaluation Conference (LREC)
·
283
PDF
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
2025
·
Penedo, Guilherme
, Kydlíček, Hynek
, et al.
·
arXiv (Cornell University)
·
264
PDF
CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs
2020
·
Ahmed El-Kishky
, Vishrav Chaudhary
, et al.
·
Conference on Empirical Methods in Natural Language Processing (EMNLP)
·
263
PDF
Demystifying CLIP Data
2023
·
Xu, Hu
, Xie, Saining
, et al.
·
International Conference on Learning Representations
·
261
OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
2023
·
Paster, Keiran
, Santos, Marco Dos
, et al.
·
International Conference on Learning Representations
·
214
N-gram Counts and Language Models from the Common Crawl
2014
·
Christian Buck
, Kenneth Heafield
, et al.
·
Language Resources and Evaluation Conference (LREC)
·
209
PDF
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
2025
·
Dan Su
, Kezhi Kong
, et al.
·
Proceedings of the conference - Association for Computational Linguistics. Meeting
·
203
PDF
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases
2022
·
Louis Martin
, Angela Fan
, et al.
·
Language Resources and Evaluation Conference (LREC)
·
202
PDF
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus
2021
·
Alexandra Luccioni
, Joseph Viviano
·
181
PDF
A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal
2020
·
Demian Gholipour Ghalandari
, Chris Hokamp
, et al.
·
Proceedings of the conference - Association for Computational Linguistics. Meeting
·
161
PDF
A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl
2024
·
Stefan Baack
·
ACM Conference on Fairness, Accountability, and Transparency (FAccT)
·
152
Dated Data: Tracing Knowledge Cutoffs in Large Language Models
2024
·
Cheng, Jeffrey
, Marone, Marc
, et al.
·
arXiv (Cornell University)
·
148
PDF
Show more works
View as search
Works count:
516
Citation count:
19,320
Year
Topic
Type