Test Your Robots.txt File To Ensure The Search Engines Can Access Your Site


Check Your Robots.txt File

1st Check That You Have The File Installed.

In The Address Bar For Your Favorite Web Browser Enter Your Website address the add /robots.txt to the end.  Example http://www.seo-911.com/robots.txt

You should see:


What you don’t want to see is:


The above returns a 404 page error, which says the robots.txt file does not exit.

Confirm Your Robots.txt File Is Working

Assuming You:

  • Want to be found in the search results
  • Want Your website to be indexed by the search engine

Use the following link to test your robots.txt file:


How To Read Your Robots.txt File

When yo use the “Disallow:”  That is saying that what ever would come next is not to be read by the search engines.  So if you had a ” / ” after the “Disallow” it would tell the robots NOT to crawl your site.  If you had a folder you wanted to keep private you would have ” /private-folder ”

In this the nothing is being restricted.

how to read the robots txt file

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>