Yahoo! Slurp is the Yahoo! web indexing robot. The Yahoo! Slurp web crawler collects documents from the Web to build a searchable index for search services using the Yahoo! Search engine. These documents are discovered and crawled because other webpages contain links to these documents.

As part of the crawling effort, the Yahoo! Slurp crawler takes robots.txt standards into account to ensure that we do not crawl and index content from pages that you do not want to be included in Yahoo! Search results. If a page is disallowed to be crawled by robots.txt standards, Yahoo! does not read or use the contents of that page.

Note: The URL of a disallowed page may be included in Yahoo! Search as a "thin" document with no text content. Links and reference text from other public webpages may provide identifiable information about a URL and may be indexed as a part of web search coverage.