THE COMPANY QUIETLY FUNNELING PAYWALLED ARTICLES TO AI DEVELOPERS

Summary from the TLDR AI newsletter

At the heart of AI web scraping is Common Crawl, a small nonprofit that supplies, and is funded by, major AI labs with internet data while flagrantly disregarding paywalls and removal requests. The organization has gathered millions of articles from The New York Times, The Atlantic, and other publishers (many of them now suing AI companies), but has avoided compliance by making its search tool return "no captures" for domains that actually contain thousands of articles.