Cloudflare has officially removed Perplexity AI from its Verified Bots list and is now blocking all of its crawlers — including stealth bots — from accessing websites. The decision came after several users complained that Perplexity was violating website crawling rules, and an investigation confirmed that the company was using aggressive tactics to get around restrictions.
What Is Cloudflare’s Verified Bots Program?
Cloudflare runs a Verified Bots program that allows trusted crawlers to access websites that are protected by its services. These bots must follow certain rules, such as respecting the robots.txt file — a standard way for websites to tell bots which parts of the site should be off-limits.
To stay verified, bots must also use clear and approved IP addresses that identify them properly.
What Perplexity Did Wrong
According to Cloudflare, Perplexity broke multiple rules. It ignored robots.txt files, used rotating IP addresses, switched ASNs (Autonomous System Numbers), and spoofed user agents to sneak past blocks.
Instead of identifying itself as “PerplexityBot” or “Perplexity-User,” the crawler reportedly disguised itself as a human browser using this user agent string:
“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36”
This kind of behavior — pretending to be a regular user to avoid detection — is known as stealth crawling.
Cloudflare’s Statement on the Ban
Cloudflare confirmed that Perplexity was delisted and will be blocked going forward. In their words:
“The Internet as we have known it for the past three decades is rapidly changing, but one thing remains constant: it is built on trust. There are clear preferences that crawlers should be transparent, serve a clear purpose, perform a specific activity, and, most importantly, follow website directives and preferences. Based on Perplexity’s observed behavior, which is incompatible with those preferences, we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling.”
Key Takeaways
- Perplexity violated Cloudflare’s Verified Bots policy by not following robots.txt and using hidden IPs.
- It used rotating ASNs and spoofed user agents to sneak back into websites that tried to block it.
- Cloudflare has blocked Perplexity and updated its system to stop stealth crawling in the future.
What Site Owners Should Know
If you want Perplexity to crawl your website, and you’re using Cloudflare, you may need to check your settings. By default, Cloudflare is now blocking Perplexity’s access.
Perplexity Responds
Perplexity responded to the controversy by accusing Cloudflare of misrepresenting how its AI Assistant works. In a rebuttal, the company said:
“When companies like Cloudflare mischaracterize user-driven AI assistants as malicious bots, they’re arguing that any automated tool serving users should be suspect—a position that would criminalize email clients and web browsers, or any other service a would-be gatekeeper decided they don’t like.
This controversy reveals that Cloudflare’s systems are fundamentally inadequate for distinguishing between legitimate AI assistants and actual threats. If you can’t tell a helpful digital assistant from a malicious scraper, then you probably shouldn’t be making decisions about what constitutes legitimate web traffic.”
