FRONTIER QUALITY AT 1% OF THE COST: A PRODUCTION BENCHMARK OF LLM CLASSIFIERS

Summary from the TLDR Dev newsletter

Airbnb fine-tuned a model with 2 billion effective parameters to match frontier models on a production policy classifier, reaching 0.902 F1 with 35 ms median latency. At thousands of requests per minute, serving it on one GPU cost roughly 1% as much as frontier APIs, though hosted models were cheaper at low traffic.