PRETRAINING THE SAME MODEL WITH 16 DIFFERENT TOKENIZERS

Summary from the TLDR AI newsletter

A weird fact of modern language modeling is that we train a tokenizer first before training the model. The second weird fact is that vocab size doesn't seem to matter too much at large scales.