How do I check that my robots.txt file is working as expected?

The robots.txt analysis tool reads the robots.txt file in the same way Googlebot does. If the tool interprets a line as a syntax error, Googlebot doesn’t understand that line. If the tool shows that a URL is allowed, Googlebot interprets that URL as allowed.

This tool provides results only for Google user-agents (such as Googlebot). Other bots may not interpret the robots.txt file in the same way. For instance, Googlebot supports an extended definition of the standard. It understands Allow: lines, as well as * and $ pattern matching. So while the tool shows lines that include these extensions as understood, remember that this applies only to Googlebot and not necessarily to other bots that may crawl your site.

If a robots.txt file exists in the root directory of the domain, this tools lists the information that Google has about it, including:

• A link to the current robots.txt file on your site.
• When Google last downloaded the file – if you’ve made changes to the file after this date and time, our cached version won’t reflect the changes
• The status of the file – the HTTP response we received when we tried to downloaded it (If the status is 200, then we accessed the file successfully; if the status is 404, then the file doesn’t exist. You can learn more about status codes in RFC-2616.)
• The MIME type – if the file is a type other than text, we can’t process it
• If the robots.txt is blocking access to your home page or to any Sitemaps you’ve submitted.
• If we had trouble parsing lines in the file.

To analyze a site’s robots.txt file:

• Sign into Google webmaster tools with your Google Account.
• On the Dashboard, click the URL for the site you want.
• Click Tools, and then click Analyze robots.txt.