FINEWEB: HIGHEST QUALITY WEB SCALE TEXT DATASET

Summary from the TLDR AI newsletter

Training a language model requires trillions of high quality tokens. Much of the information regarding the construction of these datasets is not published. This stellar blog post discusses different strategies for dataset cleaning and filtering. Its authors have released many of the best datasets out there for language model training.