TAKING OUT THE TRASH: GARBAGE COLLECTION OF OBJECT STORAGE AT MASSIVE SCALE

Summary from the TLDR newsletter

A common problem when working with distributed systems on top of object storage is figuring out how to remove files from object storage that had been logically deleted either due to data expiry or compaction. This post looks at how to solve this problem using a delayed queue, async reconciliation, or both. It also looks at obvious ways to solve the problem that don't work well in practice, like bucket policies and synchronous deletion. The solution presented in the post is highly scalable, cheap, and easy to reason about.