HUGE AI MODELS CAN RUN ON A CONSUMER GRAPHICS CARD BY USING SSD STORAGE

Summary from the TLDR Hardware newsletter

Mixture-of-Experts models keep growing capacity by adding more experts, but even after quantization, the full expert pool still dwarfs consumer RAM and VRAM, so local inference has to lean on SSD storage instead - a much harder problem than just having enough disk space. SSD-LLaMA builds an SSD I/O pipeline tuned specifically for fetching experts, layered with a three-tier SSD-RAM-VRAM hierarchy that dynamically decides which experts to hold where, and CPU-GPU hybrid execution balanced against the SSD's limited bandwidth, all while running every selected expert in full rather than pruning or substituting cheaper approximations.