TLDR · MISCELLANEOUS

PAY UP OR STOP SCRAPING: CLOUDFLARE PROGRAM CHARGES BOTS FOR EACH CRAWL

arstechnica.com7 min read

Summary from the TLDR newsletter

Cloudflare is experimenting with a new feature that allows content creators to charge a fee to stop AI crawlers from scraping their websites. The 'pay-per-crawl' feature is currently in a private beta - any content creator interested in joining can sign up. It allows publishers to set a price for bots to pay before scraping content. Publishers have the option to charge some bots while letting other bots scrape for free.