CLOUDFLARE AND OPENAI: MAKING AI DISCOVERY BETTER FOR EVERYONE

Summary from the TLDR DevOps newsletter

A research pilot between Cloudflare and OpenAI has enabled the discovery and indexing of hundreds of millions of new web pages. The program used signals from roughly 1% of Cloudflare's network to help ChatGPT find content that was previously unknown. These signals identify useful pages and filter out nonexistent or duplicated content, which reduces unnecessary crawling load for site owners. Cloudflare states the metadata is used for search discovery rather than training AI models.