INDEXING THE DATA LAKE FOR ONLINE POINT QUERIES

Summary from the TLDR Data newsletter

Spotify built an external index mapping keys directly to file offsets and row numbers in unmodified Parquet files, replacing distributed SQL scan chains with parallel ranged reads for lookups across petabyte-scale data lakes. Complementary write-time optimizations (per-key page flushing, ZSTD frame resets, column interleaving, covering index entries) reduce both bytes read and round trips further, making the same files serve both batch analytics and interactive queries.