BEYOND OFFSET LAG: COMPUTING TIME IN QUEUE FOR APACHE HUDI DATA LAKE PIPELINES AT PETABYTE SCALE

Summary from the TLDR Data newsletter

Twilio's Hudi pipelines store checkpoints in S3, so ordinary Kafka consumer lag misses when lake data is actually stale. This design reads the latest valid Hudi checkpoint, seeks Kafka partitions, and computes freshness from the oldest uncommitted record, turning petabyte-scale SLA drift into a breach ratio operators can alert on.