TRANSFORMERS WITHOUT NORMALIZATION

Summary from the TLDR AI newsletter

Most Transformer models are normalized with a layer norm. This can be seen as equivalent to normalizing the model to a sphere. However, if you use carefully placed tanh functions, you can eliminate layer norm while maintaining stability. This is now equivalent to normalizing to a hypercube.