Comments on: Guide to Robots.txt Usage

By: Moe Satriani

Moe Satriani — Fri, 23 Mar 2012 14:48:20 +0000

I love the webmaster tools as well. They are easy to use without risking your site. Thanks for all of the great information on robots. I am a web application developers , and I was just starting to get into these. I really learned a lot. Thanks!

By: Stefan Vervoort

Stefan Vervoort — Fri, 08 Apr 2011 23:26:45 +0000

Thank you

By: php

php — Fri, 08 Apr 2011 12:04:06 +0000

My website has got a penalty for a dublicate content. So now i use 301 redirect for domains with www and always build a rotots.txt file.

Stefan, thanks for post!

By: Stefan Vervoort

Stefan Vervoort — Fri, 26 Dec 2008 10:50:52 +0000

No, in that case it isn’t necessary. You use robots.txt to block certain pages.

By: Scooba

Scooba — Thu, 25 Dec 2008 10:50:06 +0000

Should I create a robots.txt file if I want to open all of my pages to search engines?

By: George The Web Designer

George The Web Designer — Mon, 01 Dec 2008 23:00:22 +0000

Thank you very much for the professional explanation. I didn’t create Robots.txt before but I know Google use it, so I would have to have it on the website I develop.
I’m not sure about the robot database. After all, I’m not hiding websites from anyone. More crawling would bring more benefits to them.

By: bebo skins

bebo skins — Fri, 28 Nov 2008 01:41:49 +0000

thanks for the database link really useful

By: Antoine

Antoine — Tue, 18 Nov 2008 03:21:11 +0000

Have to love webmaster tools. It can make life so much easier. Also, you don’t break your site or screw something up which is very useful.

By: Zygor

Zygor — Wed, 08 Oct 2008 20:12:39 +0000

I also recommend to register youw website in google webmaster tools so it will be easier to see how google sees your robots.txt and whether there are mistakes or not.