STABILIZING TRAINING BY PREVENTING ENTROPY COLLAPSE

Summary from the TLDR AI newsletter

Transformers, as amazing as they are, are often hard to train, with their many instabilities. One of the main issues is when an Attention matrix’s entropy collapses. This work is a way to prevent that with a simple reparametrization.