FINEPDFS: LIBERATING 3T OF THE FINEST TOKENS FROM PDFS

Summary from the TLDR AI newsletter

PDFs have been largely ignored by the open source AI community, despite being a source of high-quality data for pretraining. Hugging Face has created a new pretraining dataset of more than 3T tokens spanning over 1,000 languages extracted from PDFs. This post details how the dataset was created.