If your web server has been configured to return a friendly or non-standard error page using a regular "HTTP 200 – OK" status, our crawler does not recognize that it has received an error and might continue to request the incorrect URL on a normal refresh cycle.

Please use an "HTTP 404 – Page Not Found" error code when returning the custom URL-not-found page. Alternatively, include a robots noindex meta tag in the response page HTML so the error page content is not indexed.