Since we crawl billions of pages from the entire Web, we use a large number of systems for web crawling. Therefore, your web server might log requests from a number of different Yahoo! crawler client IP addresses. The different crawler systems are coordinated to limit the activity on any single web server. We determine a single "web server" by IP address, so if your host is serving multiple IPs, it might see higher levels of activity.
If there are directories on your web server which you do not want represented in web search results, use robot exclusion rules as described in "How do I prevent my site or certain subdirectories from being crawled?" An exclusion rule can reduce the number of pages Yahoo! Slurp reads from your server.
There is a Yahoo! Slurp-specific extension to robots.txt that allows you to set a lower limit on our crawler request rate.
You can add a "Crawl-delay: x.x" instruction, where "x.x" is a delay value between successive crawler accesses. If the crawler rate is a problem for your server, you can set the delay up to 5 or 10 or a comfortable value for your server, but we suggest you start with small values (0.5–1), and increase as needed to an acceptable value for your server. Larger delay values add more delay between successive crawl accesses and decrease the maximum crawl rate to your web server.
For example, a robots.txt rule to set a crawl-delay of 5 for Yahoo! Slurp looks like:
User-agent: Slurp
Crawl-delay: 5
A shorter delay value of 0.5 would look like:
User-agent: Slurp
Crawl-delay: 0.5
In general you should restrict total crawler activity to your server by disallowing unimportant content with a robots.txt rule in your User-agent: Slurp section. Setting a crawl-delay might limit the coverage and freshness of your content representation in Yahoo! Search results. If you do feel that a crawl-delay is necessary, use small values (0.5–1) to avoid blocking Yahoo! Slurp discovery and refresh of your key content.
If you have continuing issues regarding the frequency of access, please use the "Contact Customer Care" button below to open an inquiry form. Copy your most recent log file that lists the Yahoo! crawler activity and the URL of the host affected into the feedback area. We need this information to process your request.