Using robots.txt file to prevent search engine spidering
When a search engine spider vists a site, say http://www.YourSite.com/, first of all, it checks for YourSite.com/robots.txt. If the robots.txt file exists (you actually created one) it will look for this code.
User-agent: *
Disallow: /
Sometimes, for certain reasons such as:
sales pages
site rules
disclaimers
privacy policies
private pages
contact pages (prevent spamming)
we don’t want search engines to spider a page.
