TOKENSPEED

Summary from the TLDR DevOps newsletter

TokenSpeed is an open-source LLM inference engine designed for agentic workloads, offering TensorRT-LLM-level performance and vLLM-level usability. It is the first inference engine to separate the control plane, implemented in C++, from the execution plane, implemented in Python. The project is backed by the non-profit LightSeek Foundation with support from vendors including NVIDIA and AMD.