HOW I SOLVED A DISTRIBUTED QUEUE PROBLEM AFTER 15 YEARS

Summary from the TLDR newsletter

Reddit used a simple and scalable queue architecture to record upvotes that scaled well, but broke in all sorts of tricky ways. The company eventually moved to durable queues, which enabled failed jobs to resume from their last completed step, meaning no data was lost when there were program crashes. Durable queues work by combining task queues with durable workflows, which helps reliably orchestrate workflows of many parallel tasks. Architecturally, they resemble conventional queues, but use a persistent store as both message broker and backend.