ALIBABA CLOUD SAYS IT CUT NVIDIA AI GPU USE BY 82% WITH NEW POOLING SYSTEM

Summary from the TLDR Dev newsletter

Alibaba Cloud's Aegaeon pooling system reportedly reduced its Nvidia AI GPU usage by 82% using token-level scheduling, allowing 213 GPUs to perform the work of 1,192 in serving LLMs.