SPEEDING UP PYTORCH INFERENCE BY 87% ON APPLE DEVICES WITH AI-GENERATED METAL KERNELS

Summary from the TLDR AI newsletter

Frontier models can write optimized GPU kernels for Apple devices to speed up inference. This post discusses how researchers generated Metal kernels that were 1.87 times faster across 215 PyTorch mod duties, with some workloads running hundreds of times faster than baseline. The results show that AI can take on portions of optimizations, leaving human engineers to focus on the most complex optimizations.