PERPLEXITY IS USING STEALTH, UNDECLARED CRAWLERS TO EVADE WEBSITE NO-CRAWL DIRECTIVES

Summary from the TLDR Web Dev newsletter

Cloudflare has discovered that Perplexity, an AI-powered answer engine, is using stealth crawling techniques to bypass website restrictions. Perplexity appears to be masking its crawler identity by modifying its user agent, rotating IPs, and ignoring robots.txt files after being blocked. This behavior violates established internet norms of transparency and respect for website preferences, so Cloudflare has responded by delisting Perplexity as a verified bot and implementing rules to block this stealth crawling activity.