20× THE CI TRAFFIC WITHOUT GETTING SLOWER: HOW WE REBUILT GIT SERVING AT DATADOG

Summary from the TLDR DevOps newsletter

Datadog built an internal Git mirror service called gitretriever to handle the scale of CI code fetches across its engineering infrastructure. The system now handles more than 100 million requests per week across roughly 5,500 repositories, and served over a billion Git requests and hundreds of terabytes of data in its first four months. Traffic grew about 20 times during that period while median serve latency held at around 40 milliseconds. Moving CI fetch traffic to gitretriever cut fetch-serving CPU on the previous backend by three to four times.