Comments on: Robots.txt Tutorial

By: Emory Rowland

Emory Rowland — Tue, 06 Sep 2011 18:57:17 +0000

That’s a tough one. If it were a small site, my first inclination would be to block the specific directories and file names except for the index. I bet someone has a better answer though.

By: SEO Services

SEO Services — Tue, 06 Sep 2011 18:01:22 +0000

I Want to exclude (block) all pages of my website instead of “home (index)” page so what will best and simple technique (coding), can use in robot.txt file?……. Thanks in advance

By: Emory Rowland

Emory Rowland — Sat, 21 May 2011 04:32:23 +0000

Sure, you would just put these lines in a separate robots.txt file in the subdomain root directory.

User-agent: *
Disallow: /

By: Md Azhar

Md Azhar — Sat, 21 May 2011 04:05:12 +0000

Hello

Thanks for sharing info, Can we block the sub domain using robots.txt ?

Thanks

By: Emory Rowland

Emory Rowland — Wed, 04 May 2011 13:22:51 +0000

It’s voluntary so I’d expect them not to follow the robots.txt rules.

By: Shiva

Shiva — Wed, 04 May 2011 06:01:01 +0000

Hello,

I can understand the robots.txt file is used by major search engines. Does the spam search engine follow the robots.txt rules.

let me know !

By: paul

paul — Fri, 10 Jul 2009 16:23:58 +0000

nice bit of info. i had no idea you could control the agents visiting your site! pity neo didnt know about that in the matrix! would have made his life a whole lot easier! ;-)

By: Ralph Jones

Ralph Jones — Thu, 02 Jul 2009 11:20:56 +0000

Very interesting post on Robots.txt. I am a bit new to this so I book marked this to help me further. Indeed sometimes there could be pages that you would want the Google bot or any other to visit temporarily and this is indeed helpful.

By: Emory Rowland

Emory Rowland — Fri, 23 Jan 2009 05:57:54 +0000

This is some good info about Googlebot:

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364

By: Dsouza jhn

Dsouza jhn — Fri, 23 Jan 2009 05:39:54 +0000

can you provide some more information on google robot.