Author: Matthew Peschong
Website: www.mankatowebdesign.com

Find out exactly how to develop a robots.txt file for your web site here.

A robots.txt protocol file, or robots exclusion standard is a conventional tool to prevent web spiders and web robots (which index your website on search engines) from being able to access certain parts of your web site.

If you are developing a robots.txt file for yourself, I would highly recommend using a robots.txt validator to ensure the search engines will properly recognize the commands for indexing your web site that you have set. That way, you can be completely confident there are no syntax errors. I hope this article has helped you. Please leave a comment if you'd like, I encourage feedback. For more information check out my post entitled "Creating A Robots.txt File"

This text file should be placed in your root directory. Below are a few examples of how you can design your robots.txt file in notepad.

This example allows all robots to visit all files because the wildcard "*" specifies all robots.

User-agent: *
Disallow:

This example keeps all robots out:

User-agent: *
Disallow: /

The next is an example that tells all crawlers not to enter into four directories of a website:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/

Example that tells a specific crawler not to enter one specific directory:

User-agent: BadBot
Disallow: /private/

Placing Robots TXT In Your Meta Tag

You can also place a notice to your robots.txt file in you page’s meta tags.

To do this, simple place the following in your web site’s head tag.

< meta name="robots" content="noindex,nofollow" />

Minnesota Web Design is an award winning web design services and online marketing services company. We are dedicated to helping small to mid sized business clients create, develop, maintain, build, and maximize their business online. My name is Matthew Peschong. I attend Minnesota State University working on my Marketing Bachleors and my MBA Degree right now in my third year.

For more articles like this, visit Caffeine Marketing.com !