Bing vs. Freakonomics – Is the Bing It On Challenge Legit?
Did Bing cook the books when they claimed that their “Bing It On Challenge” revealed that search engine users prefer using Bing on a 2:1 ratio when compared to Google? That’s the assertion made yesterday by a publication posted on the Freakonomics blog, in which Ian Ayers said he conducted his own study and determined that, even in the context of the “Bing It On Challenge”, users preferred Google’s search results over Bing’s the majority of the time. Today, Microsoft is fighting back and challenging Ayers’ assertions. So, who’s right?
Good News for Microsoft Either Way?
Ayers’ first problem with the Bing It On Challenge’s “2:1” assertion is that it used too small of a sample size: 1,000. Microsoft says that 1,000 is actually a statistically relevant sample size, citing the fact that the presidential approval poll run by Gallup only contains a sample size of 1,500, and that a study cited by Ayers himself contains 1,000 as well. Ayers also complained about the fact that Bing didn’t release the hard data on how many users chose Bing and how many chose Google. Microsoft says there was no data to actually release because they didn’t track it in the first place, citing privacy and ethics concerns. Finally, Ayers suggests that Bing stacked the deck in their favor by suggesting search queries that would result in a high likelihood of the user preferring Bing’s results. However, Microsoft says that they chose the suggested queries randomly from a list of trending topics on Bing. Matt Cutts, head of search spam at Google, weighed in as well, saying that there were so many flaws in the Bing It On Challenge (“small sample size; bias in query selection; stripping out features of Google like geolocation, personalization, and Knowledge Graph; wording of the site; selective rematches”) that he never bothered to debunk it. No matter whose assertions you choose to believe, there’s great news embedded here for Microsoft either way. Even in Ayers’ study, users preferred Bing’s search results about 41% of the time. Given that Bing only holds about 17% of search market share, this indicates that plenty of Google junkies would indeed prefer Bing – and presumably make the switch permanently – if they gave it a chance.