Although the name sounds a little campy to me, what matters is performance. Unlike Cuil which has already been largely disregarded as a lame search engine, Bing looks and behaves amazingly similar to Google, except with a little more bling added (pardon the pun).

Bing has a better image and video search, which kinda reminds me of the old Google Searchmash. If you look in the Extras section on the top right corner of Bing, there are a few extra features that you might not have noticed, which varies depending on where you specify yourself to be from. To change your geo-location, just click on the country at the top right hand and you will be presented with a whole list of countries and language options.

The most number of features is currently limited to US based surfers, in which there is a Cashback savings option, meaning you can earn back money by buying from those participating merchant’s whose advertising fees are “passed back” down to you. I do not know how Microsoft intends to manage this whole cashback thing efficiently to properly credit everyone (should it really get popular), but it looks like there was an old Cashback option at the old Live.com (which obviously never took off).

And then there is a “Webmaster Center” which only seems to apply for US folks as well. I submitted a site of mine to Bing using their “Webmaster Tools” by uploading an XML file given by Bing. Very similar to Google’s Webmaster Tools except much more basic of course. There is also a (basic) Blog option which isn’t available for all countries. But like Yahoo 360° which is closing down shortly, I don’t see this going anywhere.

Notwithstanding all these quibbles, Bing is performing well, and have these in its favor:

• its search results are strikingly similar to Google’s (I wonder why?!)
• auto defaulted to become the default search engine on IE 6 and 7
• curiosity factor
• good video and image search
• the text search look and feel (and sponsored links) are nothing short of a Google clone
• mousing over the results gives additional info about the site or page
• nice wallpaper background which change every day and come with “hidden” Easter eggs

The reactions from Yahoo and the public are expected. I do not think Yahoo is going to be overtly concerned with Bing, because of its strong position in social media. But if China’s ban on Bing is any gauge of its efficiency, I think this “new upstart” should pat itself on the back.

While still very premature to think that Bing will become a strong contender to Google, but judging from its results, it looks like it could become a Google junior. But It’s Not Google, definitely.

Now that things have cooled down a little, I just took a second look at Cuil, and found it lacking in many areas. The irrelevancy of the search results is the main gripe, when you consider that Cuil claims to index even more pages than Google as a new search engine. More than 120 billion web pages indexed is a lot of results indeed, but their main problem is relevancy.

Although created by ex Google engineers, Cuil is not prepared to handle the spam problem. If Google is already having enough problems with spam in its index, can a new start up search engine overcome this issue?

Many people are complaining that the images they place next to the results are not related to the site at all. I found it somewhat amusing to see pages from free webhosting sites comprising nothing more than a keyword spammed multiple times showing up on the front page.

Cuil claims that site content is the number one factor for its algorithm, but this leaves it vulnerable to keyword spamming. Also, by not placing filters for “Bad Neighborhoods” like Google, almost any kind of garbage site/page can end up on the front page.

OK, Cuil is a new start up, and shouldn’t be expected to be even close to any of the Big Three (Google, Yahoo, MSN) for now. I somewhat like the layout of Cuil though. But hey, have you even checked out Google’s AJAX powered SearchMash? That’s the Google engine, albeit with a different look and layout

In fact bad bots not only don’t obey the robots.txt file, they also steal information or download entire pages off your website. Bad bots add to your bandwidth consumption and choke up your spam dustbin. Everyday, unknown bots crawl your sites and they dont identify themselves.

The problem is, how do we know which are good bots and which are bad bots? A wrongly used robots.txt file will hinder spidering, which is what we dont want.

Perhaps we should take a leaf out of Wikipedia, the premier content site on the Web, that we can safely assume to be the daily target of content eating bots and all other kinds of nefarious bots.

On Wikipedia.org/robots.txt we probably have a useful list of bots to exclude from our sites. I’m not totally 100% sure on this, but I feel that we can very safely exclude some of those bots listed there, especially those branded as trouble by Wikipedia! It’s a personal choice, but we can safely assume Wikipedia to be the most hit up site on the Web by both humans AND robots. I would like to hear any comments on this.

Since bad bots never obey the rules, why bother?

There is also the opposite train of thought amongst webmasters which is to just leave the robots.txt as sparse as possible. You should experiment around and see if it makes a difference.

2 things about the robots txt file –

• Don’t think of password protecting your robots.txt file. That goes against it’s very purpose. If you password protect your robots.txt file than it will not be readable by a bot and therefore, they won’t know what pages are forbidden. Accept that the robots.txt file should be human readable.

• Listing something in your robots.txt is no guarantee that it will be excluded. There are many bots that disobey the laws. The robots txt file is really quite limited in its usefulness, as I implied earlier.

Google, Ebay and Wikipedia all employ the robots txt file on their respective sites so it might be quite useful to take a look at these mega sites’ robots txt files for the educational value.

What about the bots that blatantly disobey the robots.txt file? Can we actually do something about it? Yes there is, but it does involve a fair deal of technical know-how. Here is a very good but technical write up on it.

User-agent: *
Disallow: /

Sometimes, for certain reasons such as:

• sales pages
• site rules
• disclaimers
• privacy policies
• private pages
• contact pages (prevent spamming)

we don’t want search engines to spider a page. We certainly don’t want our “thank you” sales pages to show up in the search engines, since everyone could then come and download our stuff for free. I made a mistake of not adding the robots.txt to one of my sites and now the contact page has been spidered by Google, so maybe some spam is headed for my mailbox.

We add a robots.txt into the public_html directory and specify which pages or directories should NOT be indexed. Just create a notepad file, specify which pages should not spidered, save it as robots.text and upload it. So if your thank you page is located at yoursite.com/thankyou, than you can specify this code in your robots.txt file.

User-agent: *
Disallow: /thankyou

For particular html pages, if you just want that page to be NOT indexable, the easiest way to stop search engine spidering is to add this piece of meta tag code into the first text block of a page, or somewhere within the <head> tags. If you don’t want the page to be indexed, use this code:

<META NAME=”ROBOTS” CONTENT=”NOINDEX”>

If you only dont want the links to be parsed, then use this meta tag:

<META NAME=”ROBOTS” CONTENT=”NOFOLLOW”>

If you dont want indexing AND link parsing, use this meta tag:

<META NAME=”ROBOTS” CONTENT=”NOINDEX,NOFOLLOW”>

This will prevent that particular page from being spidered in the ways you specified. Some spiders or robots do not obey the robots.txt file commands, and these are usually”bad” robots, like spam bots or content scrapers. So, having a robots.txt file helps in identifying the bad robots and blacklisting them. And here’s how to exclude particular robots/spiders in the robots.txt file from your entire site:

User-agent: BadRobotX
Disallow: /

But there are a few nifty things about it, that would probably appeal to lazy people (I’ll probably belong somewhere there too), such as not having to place your cursor in the search box query whenever you wish to query something up. Just type anything, and your term will appear in the search box and the current search term.

Another is the “more results” at the bottom of the page, just hit the space bar and the search automatically shows more search results for you.

SearchMash will display (for certain queries), some extra tabs to further help you to narrow down your search. These options are only available at the moment for certain queries, usually relating to health conditions, certain popular brands and celebrities. But I believe this feature will be expanded to cover more search terms as well.

As well as attempting to correct your query should you make a spelling mistake (just like the normal Google search), SearchMash will ask you if you wanted to search for another related keyword whenever you query a wide popular term. Example: Search “Asia” and SearchMash asks you if you were looking for “Asia map”.

However, what I like about SearchMash is the fact that the images that it comes up with are usually of a higher quality compared to the normal Google Search. Also, it allows for blog, video, wikipedia, image searching, and even Google Maps (for countries) all in one convenient spot, which is in the upper right sidebar, by just clicking the thumbnails. As SearchMash is new, I find there are a lot less spam site results appearing in the search results as opposed to the normal Google search, maybe due to a different, slimed down algorithm that isn’t targeted by search engine spammers.