FINDING OPTIMAL TOKENIZERS

Summary from the TLDR AI newsletter

Frontier AI models are typically trained on sequences of integers known as tokens. Each token refers to some sequence of bytes, and these byte sequences often correspond to common words. This post presents an algorithm that can compute an optimal tokenizer in some settings.