MEGABYTE - MILLION BYTE SEQUENCES

Summary from the TLDR AI newsletter

Transformers are not actually end to end. There is a separate training process for the tokenizer which is strange and often leads to odd performance in general. However, if we try to naively train on bytes we quickly run out of context length given the increased length of sequences. Additionally for truly multimodal problems training directly on bytes removes complex patching and tokenization schemes. This work allows models to train directly on bytes with sequence lengths up to a million bytes. Maybe this will even help nucleus sampling for rare words too!