IMPROVING FINE-GRAINED SUBWORD UNDERSTANDING IN LLMS

Summary from the TLDR AI newsletter

StochasTok randomly decomposes tokens during training: instead of always seeing "strawberry" as one unit, models encounter it split as "straw|berry," "str|awberry," or even "s|t|r|a|w|b|e|r|r|y," learning the internal structure humans naturally perceive. Models trained with this method achieve near-perfect accuracy on character counting and multi-digit math while maintaining performance on standard benchmarks.