FINDING DEAD WEBSITES

Summary from the TLDR Web Dev newsletter

Marginalia Search implemented a system to detect website availability and big changes, including ownership transfers and domain parking, using primarily HTTP HEAD requests and DNS queries. The system maintains live data, such as website status and security posture, and historical event data, storing changes as immutable JSON records. The challenges of implementing this system included scheduling tasks efficiently, handling per-domain request limits, and dealing with the complexities of HTTPS certificate validation.