30T OPEN DATASET RELEASED

Summary from the TLDR AI newsletter

Red Pajama v2 has been released. It is 2.5x larger than the dataset used to train GPT-4. There is no code data, but it does have multilingual data from 5 languages and covers an impressive breadth. Importantly, much of the deduplication work and some heuristic filtering has already been done.