Modern Street

So Microsoft officially changed their Live search to Bing a few days ago. I was never one to use MSN Live to search (ever), but when I put it through the paces, I have to say, it’s not bad at all! Although the name sounds a little campy to me, what matters is performance. Unlike [...]

A new search engine on the Web, called Cuil just launched recently amidst a lot of hype generated by thousands of blogs picking up on the story. The huge traffic flocking to try out the new search engine eventually led to a meltdown of the site’s servers. Even when I tried to use it a [...]

You might have noticed an increase in the number of bots trawling the Web these days. Some are good, some are not. Good bots obey the robots.txt file, but unfortunately most bad bots don’t. In fact bad bots not only don’t obey the robots.txt file, they also steal information or download entire pages off your [...]

When a search engine spider vists a site, say http://www.YourSite.com/, first of all, it checks for YourSite.com/robots.txt. If the robots.txt file exists (you actually created one) it will look for this code. User-agent: * Disallow: / Sometimes, for certain reasons such as: sales pages site rules disclaimers privacy policies private pages contact pages (prevent spamming) [...]

I have been using this search engine for a while and I’ll give it the thumbs up. SearchMash is a search engine owned and operated by Google and is basically a test bed for Google to test out new interfaces and other user friendly features. True to Google style, it is very plain and appears [...]