HOW LLMS SEE THE WORLD

Summary from the TLDR Web Dev newsletter

LLMs see and process text through tokenization, breaking text into smaller units before converting them into numerical representations. Common tokenization methods include BPE, WordPiece, and SentencePiece. The method of tokenization can impact costs, context limits, and model performance in areas like non-English languages and mathematical reasoning.