Yahoo! has used web ‘crawlers’ for a long time. A web crawler is an automated program that browses the World Wide Web and indexes web pages. This helps us provide relevant results in response to user searches. Yahoo! has been using a crawler (named “Slurp”) to help us build algorithmic (web) results as well as to scan the landing pages of Yahoo! Search Marketing (YSM) advertisers to help match advertisers’ ads to search queries.

Recently, Yahoo! launched another crawler (named “Yahoo!-AdCrawler”) that also will crawl YSM advertiser landing pages. This crawler is designed to help us make relevance assessments of these pages, that helps us match ads to specific search queries.

Website operators have the ability to block web crawlers (such as Yahoo! Search’s “Slurp”) from crawling their site, by using a ‘robots.txt’ file. Adding another crawler to crawl advertiser landing pages allows web site operators flexibility to control these crawlers independently.

Website operators can set separate access policies for Slurp and Yahoo!-AdCrawler. Keep in mind that Yahoo!-AdCrawler is designed to help us deliver more relevant ads to users that can translate into more business for you as an advertiser.

The Yahoo! Search crawler (Slurp) will continue to exist and index web pages as before. You can independently set access policies for your website for Slurp and Yahoo!-AdCrawler.

We hope to increase the relevancy of the ads we display to users, by evaluating the relevance of your landing pages to your ads and keywords. Landing pages that have relevant, useful and informative content related to your keywords and ad text will be considered to have better quality.

Keep in mind that Yahoo!-AdCrawler is designed to help us deliver more relevant ads to users and that can translate into more business for you.

Tips for improving your landing pages include:

• The landing page should take users directly to the product or service described in your ad.
• Make your landing page actionable by including a specific call-to-action that corresponds to your marketing objective. In other words, close the sale, encourage the user to leave contact information, or promote the availability of a download.
• Make the page easy to use. The transaction process should be short and simple, and the page should be uncluttered.
• Most importantly, make sure the landing page loads quickly.

When in doubt, put yourself in the shoes of users and ask yourself whether the landing page is really targeted to what the user is looking or searching for.

In some cases, our assessment of your landing page relevance can affect your ad quality. You can gain an understanding of your ad quality by looking at your quality index.

Landing pages with relevant, useful and informative content related to your keyword and ad text will be considered to be of higher quality and in some cases may be reflected in your quality index.

Allowing Yahoo!-AdCrawler to crawl your landing pages can potentially improve your traffic, because the crawler is designed to help us match your ads to relevant search queries.

If you choose to block Yahoo!-AdCrawler, it may impact our ability to evaluate the relevance of your landing pages to your keywords and ads. This in turn could decrease the display of your ads for relevant searches. As a result, you may experience reduced impressions and clicks, and/or an increase in cost per click and/or minimum bid requirements. An alternative to blocking Yahoo!-AdCrawler would be to set a crawl delay (see next section).

Yahoo!-AdCrawler obeys the Robot Exclusion Standard. Specifically, Yahoo!-AdCrawler adheres to the 1996 Robots Exclusion Standard (RES).

Yahoo!-Ad Crawler obeys the first entry in the robots.txt file with a User-agent containing "Yahoo!-AdCrawler".

Example 1 - Global disallow in robots.txt:

User-agent: *

Disallow: /

The above command will block all crawlers except for those that ignore global exclusions, such as Yahoo!-AdCrawler, which will be able to make relevance assessments of the landing pages on that domain and its sub-directories.

Example 2 - Yahoo!-AdCrawler disallow in robots.txt:

User-agent: Yahoo!-AdCrawler

Disallow: /

The above commands will block only Yahoo!-AdCrawler. Yahoo!-AdCrawler will not be able to make relevance assessments of the landing pages on that domain (the latter is not recommended - see FAQ #8). Disallowed documents, including slash "/" (the home page of the site), are not crawled, nor are links in those documents followed.

Example 3 - Yahoo!-AdCrawler disallow for specific directories in robots.txt:

User-agent: Yahoo!-AdCrawler

Disallow: /sub-dir

The above command will not let Yahoo!-AdCrawler navigate past any URL on the site with “sub-dir” in that path. It is not recommended to block Yahoo!-AdCrawler (see FAQ #8).

Example 4 - Yahoo! Ad-Crawler crawl delay in robots.txt:

User-agent: Yahoo!-AdCrawler

Crawl-delay: 0.5

The above command allows you to set a limit on our crawler request rate. This "delay value" increases the time between successive Yahoo! crawler activities, and lowers the access rate to your server. Yahoo! suggests that you start with small values (0.5–1), and increase the "delay value" only as needed for an acceptable and comfortable crawling rate for your server.