Yahoo!7 Slurp obeys the Robot Exclusion Standard. Specifically, Yahoo!7 Slurp adheres to the 1996 Robots Exclusion Standard (RES).

Yahoo!7 Slurp obeys the first entry in the robots.txt file with a User-agent containing "Slurp."

• If there is no such record, it will obey the first entry with a User-agent of "*".
• If it is not able to retrieve a robots.txt file, it will assume there are no restrictions for Slurp. It will keep trying to retrieve the file, and will obey it if becomes available.

Disallowed documents, including slash "/" (the home page of the site), are not crawled, nor are links in those documents followed. Yahoo!7 Slurp does read the home page at each site and uses it internally, but if it is disallowed, it is neither indexed nor followed. If a page has robots.txt standards disallowing it to be crawled, Yahoo! will not read or use the contents of that page.

Note: The URL of a disallowed page might be included in Yahoo!7 Search Technology as a "thin" document with no text content. Links and reference text from other public web pages may provide identifiable information about a URL and may be included as part of web search coverage.

Example robots.txt:

User-agent: Slurp

Disallow: /cgi-bin/

Directives are Case Sensitive
Robots directives for Disallow/Allow are case-sensitive. Use the correct capitalization to match your actual web site:

Example of capitalization:

User-agent: Slurp

Disallow: /private

Disallow: /Private

Disallow: /PRIVATE

Additional Symbols
Additional symbols allowed in the robots.txt directives include:

'*' - matches a sequence of characters
'$' - anchors at the end of the URL string

Using Wildcard Match: '*'
A '*' in robots directives is used to wildcard match a sequence of characters in your URL. You can use this symbol in any part of the URL string that you provide in the robots directive.

Example of '*':

User-agent: Slurp

Allow: /public*/

Disallow: /*_print*.html

Disallow: /*?sessionid

The robots directives above:

1. Allow all directories that begin with 'public' to be crawled.
   Example: '/public_html/' or '/public_graphs/'
2. Disallow any files or directories which contain '_print' to be crawled.
   Example: '/card_print.html' or '/store_print/product.html'
3. Disallow any files with '?sessionid' in their URL string to be crawled.
   Example: '/cart.php?sessionid=342bca31'

Note: A trailing '*' is not needed since that is the existing matching behavior for Slurp.

In the example below, both 'Disallow' directives are equivalent:

User-agent: Slurp

Disallow: /private*

Disallow: /private

Using '$'
A '$' in robots directives is used to anchor the match to the end of the URL string. Without this symbol, Yahoo! Slurp would match all URLs against the directives, treating the directives as a prefix.

Example of '$':

User-agent: Slurp

Disallow: /*.gif$

Allow: /*?$

The robots directives above:

1. Disallow all files ending in '.gif' in your entire site.
   Note: Omitting the '$' would disallow all files containing '.gif' in their file path.
2. Allow all files ending in '?' to be included. This would not automatically allow files that just contain '?' somewhere in the URL string.

Note: The '$' symbol only makes sense at the end of the string. Hence, when Yahoo! Slurp encounters a '$' symbol, it assumes the directive terminates there and any characters after that symbol are ignored.

Using Allow:
The 'Allow' tag is supported as shown in the examples above. For additional details see:

• 1996 Robots Exclusion Standard (RES)
• Yahoo! Slurp robots noindex meta-tag
• How do I keep my page from being cached in Yahoo! Search?