MEGALODON EFFICIENT TRANSFORMER PRETRAINING

Summary from the TLDR AI newsletter

Another long context paper - this time, a new architecture that uses two novel weight updating schemes. It outperforms Llama 2 on the same number of training tokens 2T. It also scales to infinite context length at inference time.