YOUR API LATENCY BENCHMARK IS LYING TO YOU

Summary from the TLDR AI newsletter

P50 response time tells you almost nothing about performance. The metric that actually matters [13] has to account for recall, grounding rate, re-query rate, and integration overhead. Fast-and-wrong is