PRUNE AND DISTILLING LLAMA 3.1 8B INTO A 4B MODEL

Summary from the TLDR AI newsletter

Nvidia has released its Llama 3.1 minitron 4B model. The model scored 16% better on MMLU compared with training from scratch by using knowledge distillation and pruning and required 40x fewer tokens.