DEEPINFER

Summary from the TLDR AI newsletter

DeepInfer is the inference framework that powers DeepSeek's large-scale production services. This repository contains a version for offline inference with the core features needed for serving at scale. It allows anyone to run DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs. A KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode are still required for a complete inference service.