THE TEXT FILE THAT RUNS THE INTERNET

Summary from the TLDR newsletter

robots.txt has kept the internet from chaos for three decades. The simple text file represents a deal between some of the earliest pioneers of the internet to respect each other's wishes and build the internet in a way that benefits everybody. It allows anyone who runs a website to tell the web who's allowed on the site and who isn't. The system used to work, but AI is now threatening to change everything. This article takes a look at the history behind robots.txt, how it works, different types of web crawlers, the tradeoff between sites and the search engineers crawling them, and how AI changes the balance of the internet.