Comments on: Using robots.txt file to prevent search engine spidering http://www.modernstreet.com/search-engines/using-robots-txt/ A Blog on and about the Web Mon, 06 Feb 2012 17:16:24 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Justine Cooper http://www.modernstreet.com/search-engines/using-robots-txt/comment-page-1/#comment-11775 Justine Cooper Wed, 27 Apr 2011 12:47:31 +0000 http://www.modernstreet.com/search-engines/using-robots-txt/#comment-11775 I have been learning about this spider business and i really dont understand it, i know that google sends "spiders" every week or month and they look at websites for certain info. Depending on the info/words you have on your webpage represents the higher you get to the Google first page, is this correct? I have been learning about this spider business and i really dont understand it, i know that google sends “spiders” every week or month and they look at websites for certain info. Depending on the info/words you have on your webpage represents the higher you get to the Google first page, is this correct?

]]>