AI CRAWLERS NEED TO BE MORE RESPECTFUL

Summary from the TLDR Web Dev newsletter

Read the Docs is experiencing a surge in abusive site crawling by AI products and services, leading to significant bandwidth costs and time spent dealing with abuse. These AI crawlers are aggressively downloading content without basic checks, resulting in repeated downloads of large files and heavy traffic from numerous IP addresses. The company has been forced to block several sources of abusive traffic and is facing financial strain due to the increased bandwidth charges.