Since we crawl billions of pages from the entire Web, we use a large number of systems for web crawling. Therefore, your web server might log requests from a number of different Yahoo! crawler client IP addresses. The different crawler systems are coordinated to limit the activity on any single web server. We determine a single "web server" by IP address, so if your host is serving multiple IPs, it might see higher levels of activity.

If there are directories on your web server which you do not want represented in web search results, use robot exclusion rules as described in "How do I prevent my site or certain subdirectories from being crawled?" An exclusion rule can reduce the number of pages [property_slurp/] reads from your server.

There is a Yahoo! Slurp-specific extension to robots.txt that allows you to set a lower limit on our crawler request rate.

You can add a "Crawl-delay: x.x" instruction, where "x.x" is a delay value between successive crawler accesses. If the crawler rate is a problem for your server, you can set the delay to a maximum of 10, but we suggest you start with small values (0.5–1), and increase as needed to an acceptable value for your server. Larger delay values add more delay between successive crawl accesses and decrease the maximum crawl rate to your web server.

For example, a robots.txt rule to set a crawl-delay of 1 for Yahoo! Slurp looks like:

User-agent: Slurp

Crawl-delay: 1

A shorter delay value of 0.5 would look like:

User-agent: Slurp

Crawl-delay: 0.5

In general you should restrict total crawler activity to your server by disallowing unimportant content with a robots.txt rule. Setting a crawl-delay may limit the coverage and freshness of your content representation in Yahoo! Search results. If you do feel that a crawl-delay is necessary, use small values (0.5–1) to avoid blocking Slurp discovery and refresh of your key content.