CHASING THAT 32 MS TAIL LATENCY IN CLICKHOUSE

Summary from the TLDR IT newsletter

Engineers at ClickHouse investigated why simple SELECT queries experienced an unexpected 32 ms tail latency after idle periods on high-RTT networks. They discovered that unbatched metadata packets sent alongside result data quickly exhausted the TCP congestion window after it had been reset to its initial size during idleness. To resolve the issue, ClickHouse implemented application-level batching for these metadata packets, effectively halving the TCP segment count and eliminating the tail latency penalty.