THE ARCHITECTURE BEHIND OPEN-SOURCE LLMS

Summary from the TLDR newsletter

In the open-weight ecosystem, teams build on each other's innovations to compound the pace of progress. This post looks at various open source models and the engineering bets that define each one. Every major open-weight model released at the frontier since 2025 uses a Mixture-of-Experts transformer architecture. Almost every 'open source' model is actually open weight - this means the trained parameters are public, but the training data and often the full training code are not.