Yahoo! Search
>
Search
>
Search
Yahoo! Slurp—Yahoo!'s Web Crawler
Why are you crawling my site? How do I prevent my site or certain subdirectories from being crawled? How does the Yahoo! Web Crawler handle redirects? How can I reduce the number of requests you make on my web site? How do I prevent you from indexing certain pages? How do I allow you to index pages, but not place them in your cache? Why am I seeing repeated download requests? Why is your crawler asking for strange URLs that have never existed on my site? Why can't I find my web pages in your search engine? How does your crawler find documents on my site? Does your crawler follow dynamic links?