Finding bot names to exclude from your robots file.
You might have noticed an increase in the number of bots trawling the Web these days. Some are good, some are not. Good bots obey the robots.txt file, but unfortunately most bad bots don’t.
In fact bad bots not only don’t obey the robots.txt file, they also steal information or download entire pages off your website. [...]
