RUNNING A 28.9M PARAMETER LLM ON AN $8 MICROCONTROLLER

Summary from the TLDR Hardware newsletter

A developer got a 28.9M parameter language model running entirely on an ESP32-S3, a microcontroller with just 512KB of SRAM. The trick, borrowed from Google's Gemma models, is called Per-Layer Embeddings: instead of loading the full network into fast memory, about 25 million parameters live in flash as a lookup table, and only the bytes needed for the current token get read out during inference, while the smaller layers stay in SRAM. After quantization, the whole model shrinks to 14.9MB, fitting comfortably on the board's flash, and it generates about 9.5 tokens per second on an $8 chip.