Comments on: Guide to Robots.txt Usage

By: Stefan Vervoort

Stefan Vervoort — Fri, 26 Dec 2008 10:50:52 +0000

No, in that case it isn’t necessary. You use robots.txt to block certain pages.

By: Scooba

Scooba — Thu, 25 Dec 2008 10:50:06 +0000

Should I create a robots.txt file if I want to open all of my pages to search engines?

By: George The Web Designer

George The Web Designer — Mon, 01 Dec 2008 23:00:22 +0000

Thank you very much for the professional explanation. I didn’t create Robots.txt before but I know Google use it, so I would have to have it on the website I develop.
I’m not sure about the robot database. After all, I’m not hiding websites from anyone. More crawling would bring more benefits to them.

By: bebo skins

bebo skins — Fri, 28 Nov 2008 01:41:49 +0000

thanks for the database link really useful

By: Antoine

Antoine — Tue, 18 Nov 2008 03:21:11 +0000

Have to love webmaster tools. It can make life so much easier. Also, you don’t break your site or screw something up which is very useful.

By: Zygor

Zygor — Wed, 08 Oct 2008 20:12:39 +0000

I also recommend to register youw website in google webmaster tools so it will be easier to see how google sees your robots.txt and whether there are mistakes or not.