Why is Slurp crawling my page?
What is Slurp?
Slurp is Yahoo's robot (or "spider") for web-page information collection.
- Slurp collects content from partner sites for inclusion within sites like Yahoo News, Yahoo Finance and Yahoo Sports.
- It also accesses pages from sites across the Web to confirm accuracy and improve Yahoo's personalized content for our users.
How does Slurp identify itself?
Web site access by Slurp is identified by the "Slurp" User-agent. The full user-agent value you will see in your access logs is:
Preventing Slurp from reading pages on your site
Slurp obeys the Robot Exclusion Standard. If you'd like to prevent Slurp from reading some portion of your site, create a /robots.txt file and add a rule for "User-agent: Slurp".
Example of code in a robots.txt file:
Note: Slurp will obey the first entry in the robots.txt file with a User-Agent containing "Slurp". If there is no such record, it will obey the first entry with a User-Agent of "*".
Webmasters can contact Yahoo Support directly by emailing us at: firstname.lastname@example.org.