Yahoo! Slurp has support for frames and makes an effort to crawl complex URLs such as those generated by forms, content generation systems, and dynamic page generation software.

We strive to crawl sites without requiring any special support from web masters, however, there are many ways to unintentionally build sites that can not be crawled by Yahoo! Slurp or any other web search engine robot. Be sure to apply robot exclusion rules for pages or directories that are not to be crawled, such as product order pages or bulletin boards.

For more information on using robot exclusion rules see FAQs:

• How do I prevent certain subdirectories from being crawled?


• How do I prevent you from indexing certain pages?

Our Site Guidelines offer advice on practices that can result in pages not getting indexed, along with common sense recommendations for building quality web sites.