NEW LANGUAGE MODEL OBJECTIVE TRAINS FASTER WITH TIED EMBEDDINGS

Summary from the TLDR AI newsletter

Typically with language models, the input and output embedding layers are tied together. However, if you tie the input embedding to other embeddings within the model, and use a contrastive loss, you can get dramatically improved performance both in wall clock and final benchmark accuracy.