Perplexity AI has been accused of ignoring the Robots Exclusion Protocol and scraping content from websites that have explicitly denied access. The company’s “answer engine” works by crawling the web and creating a database of content from web pages. However, recent reports suggest that Perplexity has been accessing off-limits sites.
In response to these allegations, Perplexity’s CEO, Aravind Srinivas, stated that the company is not ignoring the Robots Exclusion Protocol. He clarified that Perplexity relies on third-party web crawlers in addition to its own. The third-party crawler identified by Wired was not owned by Perplexity, but by a provider of web crawling and indexing services, whose name Srinivas did not disclose due to a Non-Disclosure Agreement.
Srinivas also addressed the issue of Perplexity’s answer engine closely paraphrasing articles. He suggested that the prompts used by Wired were designed to elicit such behavior and that normal users wouldn’t see similar results. He acknowledged that the tool may occasionally “hallucinate” or generate incorrect information.
Read more: www.fastcompany.com