Comments on: Finding bot names to exclude from your robots file. http://www.modernstreet.com/search-engines/finding-bot-names-to-exclude-from-your-robots-file/ A Blog on and about the Web Mon, 06 Feb 2012 17:16:24 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: John-info http://www.modernstreet.com/search-engines/finding-bot-names-to-exclude-from-your-robots-file/comment-page-1/#comment-5231 John-info Mon, 08 Mar 2010 05:02:32 +0000 http://www.modernstreet.com/search-engines/finding-bot-names-to-exclude-from-your-robots-file/#comment-5231 It is happening to my site, frequently i get lot of bots visiting my website daily, but the resource you gave in this article is going to be very useful for me, it is better to know the name of the bots from wikipedia and block those bots from entering a site. Other than wikipedia bot list ,if anyone knows the other bots post here thanks. It is happening to my site, frequently i get lot of bots visiting my website daily, but the resource you gave in this article is going to be very useful for me, it is better to know the name of the bots from wikipedia and block those bots from entering a site. Other than wikipedia bot list ,if anyone knows the other bots post here thanks.

]]> By: musical alex http://www.modernstreet.com/search-engines/finding-bot-names-to-exclude-from-your-robots-file/comment-page-1/#comment-1672 musical alex Tue, 19 Aug 2008 20:15:40 +0000 http://www.modernstreet.com/search-engines/finding-bot-names-to-exclude-from-your-robots-file/#comment-1672 I go for flat architecture with my site and whenever i can i exclude extraneous files with robots I go for flat architecture with my site and whenever i can i exclude extraneous files with robots

]]> By: WillMacc http://www.modernstreet.com/search-engines/finding-bot-names-to-exclude-from-your-robots-file/comment-page-1/#comment-203 WillMacc Sun, 22 Jul 2007 18:47:01 +0000 http://www.modernstreet.com/search-engines/finding-bot-names-to-exclude-from-your-robots-file/#comment-203 I normally do not put anything into my robots.txt file that I do not want seen. Also, I never link to anything I want to keep private on my site. Linking and putting it into the robots.txt file is the only want bots will know it's there. I rant a lot (and I mean a lot) about this on my blog - http://www.a-daily-rant.com/ Robots.txt file is just a road-map for the bad-bots out there since it really doesn't prevent the bots from accessing the site. I normally do not put anything into my robots.txt file that I do not want seen.
Also, I never link to anything I want to keep private on my site. Linking and putting it into the robots.txt file is the only want bots will know it’s there.
I rant a lot (and I mean a lot) about this on my blog – http://www.a-daily-rant.com/

Robots.txt file is just a road-map for the bad-bots out there since it really doesn’t prevent the bots from accessing the site.

]]>