Yahoo! Slurp obeys the Robot Exclusion Standard. Specifically, Yahoo! Slurp adheres to the 1996 Robots Exclusion Standard (RES).

Yahoo! Slurp obeys the first entry in the robots.txt file with a User-agent containing "Slurp."

• If there is no such record, it will obey the first entry with a User-agent of "*".
• If it is not able to retrieve a robots.txt file, it will assume there are no restrictions for Yahoo! Slurp. It will keep trying to retrieve the file, and will obey it if becomes available.

Disallowed documents, including slash "/" (the home page of the site), are not crawled, nor are links in those documents followed. Yahoo! Slurp does read the home page at each site and uses it internally, but if it is disallowed, it is neither indexed nor followed. If a page has robots.txt standards disallowing it to be crawled, Yahoo! will not read or use the contents of that page.

Note: The URL of a disallowed page might be included in Yahoo! Search as a "thin" document with no text content. Links and reference text from other public web pages may provide identifiable information about a URL and may be included as part of web search coverage.

Example robots.txt:

User-agent: Slurp
Disallow: /cgi-bin/

Directives are Case Sensitive
Robots directives for Disallow/Allow are case sensitive. Use the correct capitalization to match your website:

Example of capitalization:

User-agent: Slurp
Disallow: /private
Disallow: /Private
Disallow: /PRIVATE

Additional Symbols
Additional symbols allowed in the robots.txt directives include:

'*' - matches a sequence of characters
'$' - anchors at the end of the URL string

Using Wildcard Match: '[b]*'[/b]
A '*' in robots directives is used to wildcard match a sequence of characters in your URL. You can use this symbol in any part of the URL string that you provide in the robots directive.

Example of '**':

User-agent: Slurp
Allow: /public*/
Disallow: /*_print*.html
Disallow: /*?sessionid

The robots directives above:

1. Allow all directories that begin with "public" to be crawled.
   Example: /public_html/ or /public_graphs/
2. Disallow files or directories which contain "_print" to be crawled.
   Example: /card_print.html or /store_print/product.html
3. Disallow files with "?sessionid" in their URL string to be crawled.
   Example: /cart.php?sessionid=342bca31

Note: A trailing '*' is not needed since that is the matching behavior for Yahoo! Slurp.

In the example below, both 'Disallow' directives are equivalent:

User-agent: Slurp
Disallow: /private*
Disallow: /private

Using '$'
A '$' in robots directives is used to anchor the match to the end of the URL string. Without this symbol, Yahoo! Slurp would match all URLs against the directives, treating the directives as a prefix.

Example of '$':

User-agent: Slurp
Disallow: /*.gif$
Allow: /*?$

The robots directives above:

1. Disallow all files ending in '.gif' in your entire site.
   Note: Omitting the '$' would disallow all files containing '.gif' in their file path.
2. Allow all files ending in '?' to be included. This would not allow files that just contain '?' somewhere in the URL string.

Note: The '$' symbol only makes sense at the end of the string. Hence, when Yahoo! Slurp encounters a '$' symbol, it assumes the directive terminates there and any characters after that symbol are ignored.

Using Allow:
The 'Allow' tag is supported as shown in the examples above.

For additional details see:

• 1996 Robots Exclusion Standard (RES)
• Yahoo! Slurp robots noindex metatag
• How to Keep Your Page From Being Cached in Yahoo! Slurp.