Using robots.txt file to prevent search engine spidering
When a search engine spider vists a site, say http://www.YourSite.com/, first of all, it checks for YourSite.com/robots.txt. If the robots.txt file exists (you actually created one) it will look for this code. User-agent: * Disallow: / Sometimes, for certain reasons such as: sales pages site rules disclaimers privacy policies private pages contact pages (prevent spamming)… Read More »
