DATA CURATION VIA JOINT EXAMPLE SELECTION FURTHER ACCELERATES MULTIMODAL LEARNING

Summary from the TLDR AI newsletter

Actively selecting the next best batch is a challenging and open problem in pre-training. This work from DeepMind explores how to spend just 10% of the FLOPs and hard mine negative samples and still match SOTA for various tasks.