MAKING SMARTER 'MINI' MODELS USING KNOWLEDGE DISTILLATION

Summary from the TLDR AI newsletter

This study explores how to make 'student' networks (smaller models) learn better from 'teacher' networks (bigger models) in situations they've never seen before. It introduces a new method called Weight-Averaged Knowledge Distillation (WAKD) that does just as well as current techniques but is simpler to use.