SCALING DATA REPETITION FOR LLMS

Summary from the TLDR AI newsletter

The optimal amount of high-quality domain data repetition increased mildly with model size at a fixed tokens-per-parameter ratio. Smaller proxy models could therefore help estimate repetition schedules for larger models, with lower-loss domains generally tolerating more reuse.