Yahoo! Search
>
Search
>
Search
Yahoo! Slurp—Yahoo!'s Web Crawler
Why are you crawling my site? How do I prevent my site or certain subdirectories from being crawled? How does the [property_webcrawler/] handle redirects? How can I reduce the number of requests you make on my web site? How do I prevent you from indexing certain pages? How do I allow you to index pages, but not place them in your cache? Why am I seeing repeated download requests? Why is your crawler asking for strange URLs that have never existed on my site? Why can't I find my web pages in your search engine? How does your crawler find documents on my site? Does your crawler follow dynamic links? How do I use the rel="nofollow" attribute to help combat weblog comment spam?
Copyright © 2008 Yahoo! Southeast Asia Pte Ltd (Co. Reg. No. 199700735D). All Rights Reserved.
Privacy Policy - Terms of Service
Privacy Policy - Terms of Service