MAKING A WEBSITE AI FRIENDLY: WHAT THE BOTS ACTUALLY FETCHED FROM MINE

Summary from the TLDR Dev newsletter

An analysis of 18 days of traffic to one personal site found that verified crawlers mostly fetched ordinary pages, robots.txt, and the sitemap. Markdown copies attracted verified crawlers within a day, while llms.txt saw no verified training or search crawler requests, suggesting that basic HTML and measured traffic matter more than adding every proposed discovery file.