THE BUG THAT TAUGHT ME MORE ABOUT PYTORCH THAN YEARS OF USING IT

Summary from the TLDR Dev newsletter

This dev discovered a niche PyTorch bug while working on a sparse autoencoder, where the training loss plateaued due to the encoder weights failing to update on the Apple Silicon GPU (MPS). The root cause was traced to some of the Adam optimizer's operations silently failing to update output tensors on MPS due to the way memory was handled. A workaround was implemented by creating a contiguous temporary tensor, performing the operation, and then copying the results back to the original non-contiguous tensor.