REFINING PRETRAINING DATA PROGRAMMATICALLY

Summary from the TLDR AI newsletter

ProX is a framework that treats data refinement as a programming task, allowing models to apply fine-grained operations to individual examples at scale. It improves the quality of pre-training corpora by using small language models to generate programs.