Yahoo!'s Web Crawler

• Why are you crawling my site?
• How do I prevent my site or certain subdirectories from being crawled?
• How does the Yahoo! webcrawler handle redirects?
• How can I reduce the number of requests you make on my web site?
• How do I prevent you from indexing certain pages?
• How do I allow you to index pages, but not place them in your cache?
• Why am I seeing repeated download requests?
• Why is your crawler asking for strange URLs that have never existed on my site?
• Why can't I find my web pages in your search engine?
• How does your crawler find documents on my site?
• Does your crawler follow dynamic links?