MICRON REPORTEDLY EXPLORES NEAR-GPU NAND, A FLASH TIER SITTING BETWEEN HBM AND THE SSD

Summary from the TLDR Hardware newsletter

Micron is reportedly investigating near-GPU NAND, high-endurance flash modules placed right on the GPU package where HBM sits today, aiming to fill the massive latency and bandwidth gap between HBM and a system's SSD. The pitch is memory-bound inference: large models buy extra accelerators purely to hold weights and KV cache, an expensive way to add capacity since HBM is the priciest, tightest-supplied memory around, so a cheaper flash tier fast enough to keep compute fed could let the same model run on fewer GPUs.