EFFICIENT WEB CRAWLING FOR LLM PRE-TRAINING

Summary from the TLDR AI newsletter

Most data from web crawls gets thrown away due to poor text extraction performance. This work dramatically improves how many documents get saved and uses that to train stronger models on a fraction of the crawled data.