ABUSING SQLITE TO HANDLE CONCURRENCY

Summary from the TLDR Web Dev newsletter

SkyPilot uses SQLite for state management, but had issues with high concurrent writes when scaling its Managed Jobs feature, leading to database lock errors. After investigating SQLite's internals, it found that lock acquisition is prone to starvation due to the lack of FIFO guarantees and that WAL mode doesn't fully solve concurrent write problems. Avoiding SQLite for high write concurrency is the best option, but WAL mode and a high lock timeout can be used if you must use SQLite for such situations.