THE SECRET LIFE OF DNS PACKETS: INVESTIGATING COMPLEX NETWORKS

Summary from the TLDR Web Dev newsletter

Stripe experienced unexpected DNS errors manifested as SERVFAIL responses, impacting internal requests. Investigation revealed a backlog of reverse DNS lookups (PTR records) waiting on the VPC resolver (10.0.0.2), despite seemingly normal query volume and resource usage. Analyzing packet captures and Hadoop job logs pinpointed a single job performing numerous reverse DNS lookups, exceeding the VPC resolver's per-interface packet limit due to request retries.