FOR OVER A DECADE, WE'VE ACCEPTED THAT END-TO-END BACKPROP IS THE ONLY WAY TO TRAIN DEEP NETWORKS

Summary from the TLDR AI newsletter

Holding the entire network in memory at once is why AI training is hitting a resource wall. Sakana Labs has found a new way to break the network into blocks and train them independently. The trick was to treat the network's forward pass like a diffusion model denoising a signal. This slashes the memory needed to train deep models.