CHALLENGES AND RESEARCH DIRECTIONS FOR LARGE LANGUAGE MODEL INFERENCE HARDWARE

Summary from the TLDR AI newsletter

Large language model inference is hard. The primary challenges are memory and interconnect rather than compute. This paper highlights four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth, Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth, and low-latency interconnect to speedup communication. The paper focuses on datacenter AI, but it also reviews the architectures' applicability for mobile devices.