TLDR · PROGRAMMING, DESIGN & DATA SCIENCE
SCALING GIT AT OPENAI
Summary from the TLDR newsletter
OpenAI's monorepo's size and demands increased at a staggering pace over the last year, necessitating a revamp of the AI lab's source control system. Its technical staff built Pando, a new backend for Git repositories optimized for concurrent pushes to many branches and to maximize read throughput. The team implemented the backend without interrupting any engineering work or changing how the team uses Git. Pando cut median push latency in production by nearly 5x, and it now delivers nearly a petabyte of Git data per day.