Why is Slurp crawling my page?

Are you a webmaster looking for more info about the "Slurp" User-agent? We've got you covered.

What is Slurp?

Slurp is the Yahoo Search robot for crawling and indexing web page information. Although some Yahoo Search results are powered by our partners, sites should allow Yahoo Slurp access in order to appear in Yahoo Mobile Search results. Additionally, Slurp does the following:

  • Collects content from partner sites for inclusion within sites like Yahoo News, Yahoo Finance and Yahoo Sports.
  • Accesses pages from sites across the Web to confirm accuracy and improve Yahoo's personalized content for our users.

How does Slurp identify itself?

Website access by Slurp is identified by the "Slurp" User-agent. The full User-agent value you'll see in your access logs is:

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Prevent Slurp from reading pages on your site

Slurp obeys the Robot Exclusion Standard. If you'd like to prevent Slurp from reading some portion of your site, create a robots.txt file in the root directory (home folder) of your site and add a rule for "User-agent: Slurp".

Example of code in a robots.txt file:

User-agent: Slurp
Disallow: /cgi-bin/

Note - Slurp will obey the first entry in the robots.txt file with a User-agent containing "Slurp". If there is no such record, it will obey the first entry with a User-agent of "*".

More on meta tags and robots.txt

Manage how your website appears in Yahoo Search - See our guide to using meta tags and robots.txt with Slurp.

FAQ

Will Slurp slow down my website?

Slurp is designed to make reasonable requests that don't overburden websites. However, if you're concerned about site performance, you can restrict the pages that Slurp crawls by disallowing crawling of certain sub-directories, or by slowing the rate that Slurp crawls using a crawl-delay. Learn more about using the "disallow" and "crawl-delay" robots.txt directives with Slurp.

Get support

Webmasters can contact Yahoo Support directly by emailing us at: crawl-support@yahooinc.com.