DEBUGGING THE ONE-IN-A-MILLION FAILURE: MIGRATING PINTEREST'S SEARCH INFRASTRUCTURE TO KUBERNETES

Summary from the TLDR Web Dev newsletter

Pinterest's engineering team encountered a performance issue while migrating their internal search system, Manas, to Kubernetes, where a small percentage of search requests experienced massive latency spikes. Through a process of elimination and profiling, they discovered that the monitoring tool cAdvisor was the culprit, specifically its default collection of the container_referenced_bytes metric. This metric invasively scanned and cleared page table entries, causing memory contention with Manas' memory-intensive search indices. Disabling the problematic metric in cAdvisor resolved the latency spikes.