10 TRILLION SAMPLES A DAY: SCALING BEYOND TRADITIONAL MONITORING INFRA AT DATABRICKS

Summary from the TLDR Dev newsletter

Databricks rearchitected its monitoring infrastructure to handle over 10 trillion samples daily and five billion active timeseries by transitioning to a customized timeseries database called Pantheon, which has self-healing and efficient tiered storage. This new architecture also uses an advanced aggregation pipeline to manage metric cardinality and a Lakehouse-native platform named Hydra for high-cardinality troubleshooting and 50 times cheaper data storage.