Test Your Robots.txt File To Ensure The Search Engines Can Access Your Site

robots-txt-file

Check Your Robots.txt File

1st Check That You Have The File Installed.

In The Address Bar For Your Favorite Web Browser Enter Your Website address the add /robots.txt to the end.  Example http://www.seo-911.com/robots.txt

You should see:

robots-txt-file-102

What you don’t want to see is:

robot-txt-error-102

The above returns a 404 page error, which says the robots.txt file does not exit.

Confirm Your Robots.txt File Is Working

Assuming You:

  • Want to be found in the search results
  • Want Your website to be indexed by the search engine

Use the following link to test your robots.txt file:

http://www.frobee.com/robots-txt-check


How To Read Your Robots.txt File

When yo use the “Disallow:”  That is saying that what ever would come next is not to be read by the search engines.  So if you had a ” / ” after the “Disallow” it would tell the robots NOT to crawl your site.  If you had a folder you wanted to keep private you would have ” /private-folder ”

In this the nothing is being restricted.

how to read the robots txt file

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>