Why is your crawler asking for strange URLs that have never existed on my site?
Last Updated: July 30, 2007
Text Size: A A A

Save to My Help

Save this article to My Help for easy reference. You can visit the article at any time from any computer.

Replace an article

You have reached the maximum number of saved articles. Your oldest saved article will be replaced with the new one.

Some web servers send a site navigation page or other response page with a "HTTP 200 OK" response instead of a "HTTP 404 Not Found" result for page-not-found conditions. To check on web server handling of page-not-found conditions, Yahoo! Slurp occasionally sends deliberately odd URLs built from random words to sites from which no 404 results have been seen. These URLs are built intentionally to not match any actual content at the site. We save information on the web server response to requests for non-existent pages so we can correctly recognize and remove obsolete URLs in our search database.

A Yahoo! Slurp check for 404 results from a web server consists of requests for up to 10 such URLs. The check for 404 behavior is not a normal part of Yahoo! Slurp site refresh, so such requests are rare.

Was this information helpful?      

My Help

Forgot your ID or password?

Sign In

Sign in to see your account information saved articles and more.
  1. Recent Searches

  2. Saved Articles

    Sign in to see your account information saved articles and more.

Still Need Help?