Perplexity is allegedly scraping websites it’s not supposed to, again

Date:

Share:


Web crawlers deployed by Perplexity to scrape websites are allegedly skirting restrictions, according to a new report from Cloudflare. Specifically, the report claims that the company’s bots appear to be “stealth crawling” sites by disguising their identity to get around robots.txt files and firewalls.

Robots.txt is a simple file websites host that lets web crawlers know if they can scrape a websites’ content or not. Perplexity’s official web crawling bots are “PerplexityBot” and “Perplexity-User.” In Cloudflare’s tests, Perplexity was still able to display the content of a new, unindexed website, even when those specific bots were blocked by robots.txt. The behavior extended to websites with specific Web Application Firewall (WAF) rules that restricted web crawlers, as well.

A flowchart created by Cloudflare to illustrate the different ways Perplexity's web crawlers try to access the content of a website.

Cloudflare

Cloudflare believes that Perplexity is getting around those obstacles by using “a generic browser intended to impersonate Google Chrome on macOS” when robots.txt prohibits its normal bots. In Cloudlfare’s tests, the company’s undeclared crawler could also rotate through IP addresses not listed in Perplexity’s official IP range to get through firewalls. Cloudflare says that Perplexity appears to be doing the same thing with autonomous system numbers (ASNs) — an identifier for IP addresses operated by the same business — writing that it spotted the crawler switching ASNs “across tens of thousands of domains and millions of requests per day.”

Tech Reader has reached out to Perplexity for comment on Cloudflare’s report. We’ll update this article if we hear back.

Up-to-date information from websites is vital to companies training AI models, especially as service’s like Perplexity are used as replacements for search engines. Perplexity has also been caught in the past circumventing the rules to stay up-to-date. Multiple websites reported in 2024 that Perplexity was still accessing their content despite them forbidding it in robots.txt — something the company blamed on the third-party web crawlers it was using at the time. Perplexity later partnered with multiple publishers to share revenue earned from ads displayed alongside their content, seemingly as a make-good for its past behavior.

Stopping companies from scraping content from the web will likely remain a game of whack-a-mole. In the meantime, Cloudflare has removed Perplexity’s bots from its list of verified bots and implemented a way to identify and block Perplexity’s stealth crawler from accessing its customers’ content.



Source link

━ more like this

iPhone 17 Pro, Max and Air: Release dates, colors and everything else you need to know about Apple’s new phones

After seeing the changes coming with iOS 26, we're ready to finally see Apple's newest collection of smartphones: the iPhone 17 lineup. While...

Apple iOS 26 beta: How to download the update on your iPhone, new features like Liquid Glass and everything else you need to know

Which iPhones will be able to upgrade to iOS 26?A few iPhone models that run the current version of iOS — iPhone XR,...

Here’s a comprehensive list of compatible iPhones that can download the iOS 26 beta today

If you're a devoted iPhone owner like me, you'll be excited to check out all the exciting new updates coming to Apple's operating...

NASA’s Lunar Trailblazer mission ends in disappointment

The Lunar Trailblazer mission to the moon officially ended on July 31, but it wasn't a complete journey. NASA today that its...
spot_img