DEEPSPEED RELEASES INFERENCE SOFTWARE

Summary from the TLDR AI newsletter

This is Deepspeed's new software for competing with vLLM and text-generation interfaces for the fast serving of language models. It comes with many state-of-the-art acceleration features and preliminary results suggest it's up to 2.4x faster.