HOW MANY LABELED EXAMPLES DOES A TEXT CLASSIFIER ACTUALLY NEED? I MEASURED IT

Summary from the TLDR Data newsletter

Before defaulting to an LLM for text classification, benchmark a cheap and near-instant classical model against your available labels, expected volume, and acceptable error rate. In a small support-ticket experiment, TF-IDF accuracy rose from 40% to 60% as training data increased from 2 to 10 examples per category, but intent-based ambiguities remained, suggesting a hybrid approach where LLMs bootstrap labels or handle difficult cases while the classical model processes routine ones.