SCALING OUR OBSERVABILITY PLATFORM BEYOND 100 PETABYTES BY EMBRACING WIDE EVENTS AND REPLACING OTEL

Summary from the TLDR Web Dev newsletter

ClickHouse scaled its internal observability platform, LogHouse, to over 100 petabytes by developing SysEx, a specialized system for ClickHouse-to-ClickHouse data transfers. SysEx, unlike OTel, preserves native ClickHouse types and removes intermediate conversions, resulting in a 20x increase in event volume handled with significantly less CPU usage. They use HyperDX, a ClickHouse-native UI, for seamless data exploration, correlation, and root-cause analysis, while still using Grafana for existing dashboards.