EXPERT UPCYCLING

Summary from the TLDR AI newsletter

Amazon researchers released a method to expand Mixture-of-Experts models mid-training by duplicating and specializing experts, improving performance without increasing inference cost.