COMPUTE BETTER SPENT: REPLACE DENSE LAYERS WITH STRUCTURED MATRICES

Summary from the TLDR AI newsletter

Most compute for Transformers is spent in the Linear layers. This work builds on muP and Monarch matrices to create a structured representation that uses less compute and has better scaling laws than naive dense layers.