FRONTIER QUALITY AT 1% OF THE COST: A PRODUCTION BENCHMARK OF LLM CLASSIFIERS

Summary from the TLDR Data newsletter

Airbnb fine-tuned a model with 2 billion effective parameters to screen content for pricing-policy violations. Its F1 score rose from 0.563 to 0.902, matching the best frontier APIs, with a median latency of 35ms against their roughly 900ms. At thousands of requests a minute, the frontier models cost two orders of magnitude more than one self-hosted GPU.