EFFICIENT LLM SCHEDULING BY LEARNING TO RANK

Summary from the TLDR AI newsletter

When serving many concurrent requests to a large language model, things often get bottlenecked because we don't know how long output generation will be, so we get head of line bottlenecks. However, you can learn to rank relative lengths between requests and serve the shortest ones first, which leads to a 6.5x higher throughput for multi batch generation.