MOSAICBERT: A BIDIRECTIONAL ENCODER OPTIMIZED FOR FAST PRETRAINING

Summary from the TLDR AI newsletter

BERT is an Encoder-only Transformer. This means it is often used to represent natural language in the form of embeddings and isn’t usually used to generate text. Importantly, it was one of the first Transformers used in search because it can represent both the query and the information to be retrieved. The team at Mosaic has integrated FlashAttention, GLU, and many other innovations to dramatically improve pretraining speed (1 hour, $20) while matching the performance of much larger traditional BERT models.