TLDR IT · MISCELLANEOUS

CAN KAFKA SUPPORT ELASTIC PARTITIONING?

terroir.systems15 min read

Summary from the TLDR IT newsletter

Kafka could support elastic partitioning, dynamically splitting and merging key ranges to scale ingestion and processing while balancing load. The design could preserve per-key ordering and Kafka's batch-oriented efficiency, but it would complicate consumer state locality, repartitioning, replay tracking, and recovery. Eliminating partitions entirely would sacrifice important batching and storage benefits.