Using the Energy of Robots.txt

SEnuke: Ready for action

Sometimes, we may want search-engines not to index certain elements of the site, or even exclude other SE in the site completely.

This really is where a simple, little 2-line text file called robots.txt comes in.

Once we've a website up and running, we need to make certain that all visiting search-engines can access all the pages we want them to check out.

Sometimes, we might want search engines not to catalog certain elements of the site, or even ban other SE from the site all together.

This is where a simple, little 2 line text file called robots.txt comes in.

Robots.txt lives within your websites main directory (o-n LINUX systems that is your /public_html/ directory), and looks some thing just like the following:

User-agent: *

Disallow:

The initial line controls the bot that will be visiting your site, the second line controls if they're allowed in, or which parts of the site they are not allowed to go to

Then simple repeat the above mentioned lines, If you would like to take care of multiple spiders. We learned about per your request by searching webpages.

Therefore an example:

User-agent: googlebot

Disallow:

User-agent: askjeeves

Disallow: /

This may allow Goggle (user-agent name GoogleBot) to see every page and index, while in the same time banning Ask Jeeves in the site entirely.

To discover a reasonably up to date list of robot individual names this visit http://www.robotstxt.org/wc/active/html/index.html

Its still very advisable to place a robots.txt file on your site, even though you want to allow every robot to index every page of your site. It will end your error logs replenishing with entries from search engines attempting to access your robots.txt file that doesnt occur. Browsing To continue reading likely provides tips you might tell your father.

To find out more on robots.txt see, the total listing of sources about robots.txt at http://www.websitesecrets101.com/robotstxt-further-reading-resources.