AI CRAWLERS ARE HAMMERING SITES AND NEARLY TAKING THEM OFFLINE

Summary from the TLDR Marketing newsletter

AI crawlers are overwhelming websites with excessive traffic, slowing them down or nearly taking them offline. iFixit and Game UI Database both blocked crawlers from Anthropic and OpenAI after experiencing major performance issues. While AI companies claim to follow industry standards, website owners report disruptions and high costs. Many sites are now blocking AI crawlers using robots.txt.