Why is Slurp crawling my page?
Are you a webmaster looking for more info about the "Slurp" User-agent? We've got you covered.
What is Slurp?
Slurp is the Yahoo Search robot (or crawler) for webpage information collection. Here are the additional tasks Slurp has:
- Slurp collects content from partner sites for inclusion within sites like Yahoo News, Yahoo Finance and Yahoo Sports.
- It also accesses pages from sites across the Web to confirm accuracy and improve Yahoo's personalized content for our users.
How does Slurp identify itself?
Website access by Slurp is identified by the "Slurp" User-agent. The full User-agent value you'll see in your access logs is:
Prevent Slurp from reading pages on your site
Slurp obeys the Robot Exclusion Standard. If you'd like to prevent Slurp from reading some portion of your site, create a /robots.txt file and add a rule for "User-agent: Slurp".
Example of code in a robots.txt file:
Note - Slurp will obey the first entry in the robots.txt file with a User-agent containing "Slurp". If there is no such record, it will obey the first entry with a User-agent of "*".
More on meta tags and robots.txt
Manage how your website appears in Yahoo Search - See our guide to using meta tags and robots.txt with Slurp.
Webmasters can contact Yahoo Support directly by emailing us at: firstname.lastname@example.org.
Thank you! Your feedback has successfully been submitted.
Please tell us why you didn't find this helpful.