BETTER PROMPTING AND FASTER INFERENCE WITH SGLANG AND RADIX ATTENTION

Summary from the TLDR AI newsletter

LMSYS has released two new innovations for language model inference. The first is a backend change which improves overall tokens per second performance. The second is an embedded domain-specific language for complex prompting techniques which allows for prompting parallelism.