GOOGLE PROPOSES RANDOM MASKING FOR LLM OPTIMIZATION

Summary from the TLDR AI newsletter

Google researchers showed that randomly masking parameter updates can outperform dense adaptive optimizers, with a masked RMSProp variant exceeding recent state-of-the-art methods. Their proposed Magma optimizer uses momentum-aligned gradient masking to improve pre-training efficiency.