FISHING FOR MAGIKARP: AUTOMATICALLY DETECTING UNDER-TRAINED TOKENS IN LARGE LANGUAGE MODELS

Summary from the TLDR AI newsletter

Language models rely on separately trained tokenizers. These may produce tokens that are never seen during language model training. There are many in even the most powerful modern language models. This paper explores this phenomenon and presents their findings for identifying and dealing with these tokens.