REPHRASING THE WEB

Summary from the TLDR AI newsletter

“Up captioning” is an idea used in many AI systems to improve labels while training. This work from Apple applies this to pre-training by rephrasing C4 as Q/A pairs, instructions, and more. The study found the rephrasing step sped up convergence by 10x, making the model dramatically more sample efficient - at the cost of the rephrasing step.