Archive for November, 2007
How do I check that my robots.txt file is working as expected?
Mardian Purwanto - Nov 17th, 2007
The robots.txt analysis tool reads the robots.txt file in the same way Googlebot does. If the tool interprets a line as a syntax error, Googlebot doesn’t understand that line. If the tool shows that a URL is allowed, Googlebot interprets that URL as allowed. This tool provides results only for Google user-agents (such as Googlebot). [...]
Read More..>>