Yahoo! Search
>
Yahoo! Search
Help
>
Webcrawler
Yahoo! Slurp—Yahoo!'s Web Crawler
Why are you crawling my site? How do I prevent my site or certain subdirectories from being crawled? How does the Yahoo! Web Crawler handle redirects? How can I reduce the number of requests you make on my web site? How do I prevent you from indexing certain pages? How do I allow you to index pages, but not place them in your cache? Why would I see repeated download requests? Why is your crawler asking for strange URLs that have never existed on my site? Why can't I find my web pages in your search engine? How does your crawler find documents on my site? Does your crawler follow dynamic links? How do I use the rel="nofollow" attribute to help combat weblog comment spam?