TLDR Marketing · MISCELLANEOUS

HOW CLOUDFLARE IS MAKING AI PAY FOR CONTENT

blog.bytebytego.com9 min read

Summary from the TLDR Marketing newsletter

Cloudflare is turning the reverse proxy position it holds in front of a large share of the web into a payment counter for AI traffic. Its Pay Per Crawl tool lets a site block, allow, or charge a flat fee per crawl, but more than half of crawls from well-behaved bots just re-fetch pages that haven't changed, so payment is shifting from the crawl to the use. Web Bot Auth verifies a caller's identity with cryptographic signatures, and x402 settles payment through the HTTP 402 status code before the origin ever responds.