POWERINFER FOR LLMS ON CONSUMER GPUS

Summary from the TLDR AI newsletter

This paper explores the idea that models have hot neurons used for almost all inputs and cold neurons that are used much less often. By preloading the hot neurons to the GPU, you can save memory without sacrificing much throughput. A companion code library is available.