COLIBRÌ

Summary from the TLDR DevOps newsletter

Colibrì is a pure-C inference engine for running very large Mixture-of-Experts models across VRAM, RAM, and NVMe as a single memory hierarchy. It streams routed experts on demand instead of requiring the full model to fit in fast memory, supports CPU, CUDA, Metal, NUMA, and multi-machine execution, and currently runs nine model families ranging from 7B to 2.8T parameters.