When a webpage changes, the changes are reflected in our database the next time the page is crawled and indexed by our web crawler. Our web crawler revisits known webpages periodically to make sure that the pages are still there and to update the database information.

If you add new pages to an existing site, make sure to build links from your existing webpages and navigation tools to the new pages. When our crawler refreshes content from the known pages, it finds the links to the new pages and considers them for addition to the web database.

Our crawler is not always able to follow dynamic links, so to make sure that content is included, it is best to provide a static link to new content. If your site navigation is normally done only with dynamic links, you can create a site map page with a static link map of your site so that our crawler can discover all of your content.