VECTOR SEARCH ON APACHE ICEBERG WITHOUT MOVING SOURCE DATA

Summary from the TLDR Data newsletter

StarTree adapts Pinot's HNSW vector index for Iceberg tables, keeping Parquet source records in object storage while storing and serving a derived index alongside them. This enables hybrid similarity and SQL-filtered queries without a second copy of the records. The trade-off: indexes are roughly as large as the embeddings, and freshness follows Iceberg snapshot synchronization rather than real-time ingestion.