AI SITE PERPLEXITY USES “STEALTH TACTICS” TO FLOUT NO-CRAWL EDICTS

Summary from the TLDR InfoSec newsletter

A recent report from Cloudflare stated that Perplexity is deliberately circumventing robots.txt and utilizing evasion techniques to avoid web application blocks. The researchers identified that when Perplexity's crawlers encounter Cloudflare's blocks or a robots.txt file, they cycle through a list of IPs not in Perplexity's published IP ranges and shuffle ASNs to avoid blocks. Other sites such as Forbes, Reddit, and Wired have also leveled similar accusations at Perplexity.