AI AND THE MEMORY WALL

Summary from the TLDR AI newsletter

The rise of LLaMA.cpp was surprising to many. Why can an ordinary MacBook run state-of-the-art models only a bit slower than a mighty A100? The answer is that when you're only doing a single batch, one token at a time, memory bandwidth is the bottleneck and the gap between the M2's memory bandwidth and the A100's is not that large.