LIBRARY FOR DATA FILTERING

Summary from the TLDR AI newsletter

HuggingFace has released Datatrove, a library for filtering large text datasets. It has many useful primitives and full parallel pipelines for text filtering. Examples using Gopher quality filters in C4 are available.