HITTING A BILLION TOKENS PER MINUTE ON ONE GPU BY COMBINING A QUERY PLANNER AND AN INFERENCE ENGINE

Summary from the TLDR AI newsletter

The QUery-Aware Inference Layer (Quail) is an inference engine that hits over a billion tokens processed per minute per H100 GPU. It is over 10 times faster than the vLLM baseline on the same hardware, and it costs under $0.06 per billion tokens on Modal. This post provides a quick overview of how Quail works. It focuses more on considerations for inference engineers.