CUTTING OUR INFERENCE BILL BY 68% BY BEING OBSESSIVE ABOUT OPTIMIZATION

Summary from the TLDR newsletter

Cloudflare lowered its average review cost from $2.43 to $0.77. The biggest single saving came from checking what actually reached the provider on each session's first request, and finding a way to order those messages in a way that was as consistent as possible across commits, merge requests, and repositories. Check whether each session's first request actually reads from the cache and whether reviewers still find real problems at the effort level being paid for.