HOW A 3RD PARTY API CAN RUIN YOUR WEEKEND

Summary from the TLDR Web Dev newsletter

This is a story of how a critical microservice that depends on a 3rd-party API failed and caused a disruption for over two weeks. The API suddenly failed one day, and because the handling of API events was not done in real time, the failed events piled up, reaching the API limit and causing many events to be dropped. The post dives into the lessons learned from the incident, such as automating re-runs of failed jobs, involving other teams more, and establishing an on-call rotation earlier.