Why is Slurp crawling my page?

Are you a webmaster looking for more info about the "Slurp" User-agent? We've got you covered.


What is Slurp?

Slurp is Yahoo's robot (or "spider") for web-page information collection.

  • Slurp collects content from partner sites for inclusion within sites like Yahoo News, Yahoo Finance and Yahoo Sports.
  • It also accesses pages from sites across the Web to confirm accuracy and improve Yahoo's personalized content for our users.

How does Slurp identify itself?

Web site access by Slurp is identified by the "Slurp" User-agent. The full user-agent value you will see in your access logs is:

Mozilla/5.0 (compatible; Yahoo Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Preventing Slurp from reading pages on your site

Slurp obeys the Robot Exclusion Standard. If you'd like to prevent Slurp from reading some portion of your site, create a /robots.txt file and add a rule for "User-agent: Slurp".

Example of code in a robots.txt file:

User-agent: Slurp
Disallow: /cgi-bin/

  Note: Slurp will obey the first entry in the robots.txt file with a User-Agent containing "Slurp". If there is no such record, it will obey the first entry with a User-Agent of "*".

Getting support

Webmasters can contact Yahoo Support directly by emailing us at: crawl-support@yahoo-inc.com.

  • Was this article helpful?

  • Yes
  • No

Still need help?