HOW OPENAI DELIVERS LOW LATENCY VOICE AI FOR 900M USERS

Summary from the TLDR Dev newsletter

OpenAI serves WebRTC voice AI to 900M weekly users by splitting the stack into a stateless edge relay for packet routing and a stateful transceiver that owns the heavy WebRTC state, sidestepping the port-exhaustion and state-stickiness problems WebRTC hits on Kubernetes. The key method is encoding routing metadata into the ICE ufrag, a field already exchanged during setup, so the relay routes the first packet by reading the initial STUN request rather than doing a database lookup.