TLDR DevOps · MISCELLANEOUS

FIXING TOOL-CALL ACCURACY IN VLLM

ubicloud.com7 min read

Summary from the TLDR DevOps newsletter

vLLM 0.31 can achieve 100% tool-call schema accuracy for Kimi and DeepSeek models through constrained decoding, a technique that guarantees schema adherence during token generation. This fixes a reliability issue where vLLM 0.28 produced invalid tool calls 8% of the time, causing agentic workloads to build up nonsensical context.