NORMALIZED REPRESENTATIONS ON THE HYPERSPHERE

Summary from the TLDR AI newsletter

Normalizing the internal representation of language models is the target of a lot of resource activity. This work shows that if you put every internal vector on a hypersphere, it dramatically decreases convergence time for reasonably sized models.