WHY DEEPSEEK IS CHEAP AT SCALE BUT EXPENSIVE TO RUN LOCALLY

Summary from the TLDR AI newsletter

Models that are Mixture-of-Experts with many layers, like DeepSeek, require large batch sizes and high latency otherwise throughput drops off a cliff. This is why it is commonly said that DeepSeek can't be easily run for personal use - with a single user running one inference at a time, it runs at a very low efficiency/throughput. This article explains this phenomenon in detail, as well as why some AI models are slow to respond but fast once they get going and the other effects of throughput, latency, and batch sizes.