BRANCH-TRAIN-MIX: MIXING EXPERT LLMS INTO A MIXTURE-OF-EXPERTS LLM

Summary from the TLDR AI newsletter

This work shows that you can train models individually and then merge them together into a single Mixture-of-Experts model.