HOW AIRBNB BUILT A KEY-VALUE STORE FOR PETABYTES OF DATA

Summary from the TLDR Web Dev newsletter

Airbnb's derived data infrastructure evolved from HFileService, a read-only batch processing system, to Nebula, which combined batch and real-time data but suffered from inefficiencies and complexity. To overcome these challenges, Airbnb developed Mussel, a scalable, low-latency key-value store using technologies like Apache Helix, Kafka, and HRegion to handle real-time and batch data seamlessly. Mussel now supports over 130TB of data with high availability and sub-8ms read latency.