TRANSFORMER-LITE: LLMS ON MOBILE PHONE GPUS

Summary from the TLDR AI newsletter

Running language models on phones is challenging due to latency, bandwidth, and power constraints. Using quantization, removal of the kv cache, and other optimizations - this research shows how to get 30 tokens/second generation for the powerful Gemma 2B model, which is approximately 3x faster than other frameworks.