How does the Yahoo! Web Crawler handle redirects?
Last Updated: May 13, 2009
Text Size: A A A

Save to My Help

Save this article to My Help for easy reference. You can visit the article at any time from any computer.

Replace an article

You have reached the maximum number of saved articles. Your oldest saved article will be replaced with the new one.

When one web page redirects to another web page, Yahoo! Web Search sometimes indexes the page content under the URL of the entry or "source" page, and sometimes index it under the URL of the final, destination, or "target" page.

Yahoo! Web Crawler recognizes standard HTTP result codes. The basic types of redirect recognized are:

  • 301: Permanent Redirect
  • 302: Temporary Redirect
  • META Refresh: <meta http-equiv="refresh" content=...> is recognized as a 301 if it specifies little or no delay or as a 302 if it specifies noticeable delay.

Yahoo! Web Search General Guidelines for Indexing Redirects
The examples below illustrate when Yahoo! Web Search normally records the "target" page URL or indexes the "source" page URL.

When a page in one domain redirects to a page in another domain, Yahoo! records the "target" URL.

When a top-level page in a domain presents a permanent redirect to a page deep within the same domain, Yahoo! indexes the "source" URL.

When a page deep within a domain presents a permanent redirect to a page deep within the same domain, Yahoo! indexes the "target" URL.

When a page in a domain presents a temporary redirect to another page in the same domain, Yahoo! indexes the "source" URL.

Yahoo! Web Search indexes URLs that redirect according to the general guidelines outlined above with the exception of special cases that might be read and indexed differently.

Because of mapping algorithms directing content extraction, Yahoo! Web Search is not always able to discard URLs that have been seen as 301s, so web servers might still see crawler traffic to the pages that have been permanently redirected. The Yahoo! Slurp crawler does follow redirects, which are handled as described above.

Was this information helpful?      

My Help

Forgot your ID or password?

Sign In

Sign in to see your account information saved articles and more.
  1. Recent Searches

  2. Saved Articles

    Sign in to see your account information saved articles and more.

Still Need Help?