LLM INFERENCE ON THE APPLE NEURAL ENGINE

Summary from the TLDR Dev newsletter

MacPaw's tests of LFM2.5-2.6B across three Macs found Neural Engine inference used 2.6 to 3.4 times less energy than MLX 8-bit with speculative decoding on typical requests, while long contexts benefited from splitting work with the GPU.