MINIMA, A NEW EFFICIENT LANGUAGE MODEL

Summary from the TLDR AI newsletter

This project tackles the challenge of distilling large language models into smaller student models. The researchers discovered a consistent optimal capacity gap for different student models and used the insight to create MiniMA, a 3B model distilled from a 7B teacher model. MiniMa sets new benchmarks in efficiency and performance.