Daily Edition · Page 4

THE DAILY INDEX

Technology, AI, software, security, crypto and business

ARTICLES & TUTORIALS

TLDR Web Dev

FINDING DEAD WEBSITES

Marginalia Search implemented a system to detect website availability and big changes, including ownership transfers and domain parking, using primarily HTTP HEAD requests and DNS queries. The system maintains live data, such as website status and security posture, and historical event data, storing changes as immutable JSON records. The challenges of implementing this system included scheduling tasks efficiently, handling per-domain request limits, and dealing with the complexities of HTTPS certificate validation.

TLDR IndexPage 4