A LITTLE HELP GOES A LONG WAY: EFFICIENT LLM TRAINING BY LEVERAGING SMALL LMS

Summary from the TLDR AI newsletter

DeepMind shows in this work how you can use a small language model to provide soft supervision labels and select informative/challenging data points for pretraining. This small helper model can dramatically speed up pre-training time.