SHORTER TOKENS ARE MORE LIKELY

Summary from the TLDR AI newsletter

Large language models may be pushed to select 'slop' words because those words have more possible endings, even if none of those endings are the best one. Small and seemingly unimportant changes to tokenizers can have an unexpectedly large effect on outputs. Character-level transformers may be worse, but models that try to group tokens so that they're equally 'surprising' might make it better.