BLOCKDIFF: HOW WE BUILT OUR OWN FILE FORMAT FOR VM DISK SNAPSHOTS

Summary from the TLDR Dev newsletter

Cognition AI built Blockdiff to get fast block-level diffs and snapshots of VM disks for its AI software engineer, Devin. Blockdiff addresses the slow snapshot times experienced with EC2 by creating compact, instantaneous snapshots with zero overhead by only storing the differing blocks between a base image and VM disk. The tool uses Linux filesystem concepts like sparse files and copy-on-write to efficiently serialize the diffs.