Usually when Tomi Ahonen starts repeating himself, I make a separate post about it. That’s true this time too. It’s about two surveys regarding Nokia Lumia 900. This is almost plain copy-paste from my post about Nokia’s Q2 figures so nothing new in a sense, just wanted to have a good point where to guide people.

Without further introduction, Tomi Ahonen:

the independent survey of Lumia owners by Yankee Group found that of Lumia owners in the USA, 4 out of 10 rate the Lumia as literally the worst phone possible (rating it a 1 out of 5 where 1 is worst and 5 is best).” [1]

There is one other study, a genuinely unbiased neutral survey of the same issue, by Yankee Group – which found the total opposite – 41% of specifically American Lumia owners rated it such utter rubbish smartphone, that they gave it a rating of 1 (worst) on a scale of 5 to 1. Four out of ten Lumia owners in America right now, rate their smartphone the worst thing they’ve ever seen.” [2]

Okay, results are sounding quite bad. Especially if 41% of respondents are giving the worst rating available. But let’s try to get more details: Yankee survey only used 111 people for results. So basically one person is responsible for 1% of the result. Tomi was recently laughing in Twitter for something that used 50 people for the scope of survey. Apparently 111 is fine, then?

This small part that Tomi leaves out is rather interesting: “On a 1-5 rating scale, 41% of survey participants scored the handset as a ’1′, with an average score of only 2.5 out of 5.” [3]

So… We have average of 2.5 (in scale of 1 to 5) where 41% already voted for 1? That means the remaining 59% have voted average of… 3.5!
I’ll put it this way: If 41% voted for 1, it is unrealistic to assume that nobody votes for 2. And most likely very few voted for 5, right? So let’s say 28% voted for ‘2’ (as in this kind of survey we should see some kind of gauss curve, even if the peak is in ‘1’). So 28% for that. Guess what next? In that case EVERY SINGLE respondent left voted for 5, with no 3′s or 4′s in between!
In any spread we try, there must be:
either some pile-up in the HIGH end of results OR
-very high concentration of 1′s compared to otherwise flat results.

First option is possible, explanation would be that people participating the survey were divided to two groups; those loving the device and those hating it. Like asking a random group of people what they think of a presidential candidate, you get 5’s from the members of the same party and 1’s from the others. Could be. But then this does not mean that 41% rate it bad, it means 59% rate it good. (Not saying excellent, that would need average score of 3 instead of 2.5)

Latter option is unrealistic. Now Tomi said it right: “It sounds like those communist Eastern European ‘elections’ where the President always got 95% or more of the total vote“. [2] Have you read about Russian election spread of percentages? It’s about odd pile-ups you get if you don’t have authentic take of people for your statistics. [4] And Yankee survey has very small take of people. That could be easily selected so that you get results you wish to get. Fits to picture.

Nevertheless, considering multiple positive reviews of Lumia line, I have to wonder why this one is so far away? I cannot find a single place where we would be told WHO ordered the survey and paid for it. Customer may have wanted to remain anonymous, sure. But trust me, Yankee group is not a charity organization. And I do admit it is odd to hear the next text from the mouth of someone who just did “genuinely unbiased neutral survey”:

Yankee Group’s Howe argued that Microsoft should buy Nokia outright if it wants to get its mobile offerings into the mainstream, especially now that Google has completed its acquisition of Motorola.” [5]

Now to couple this up, we have another survey. Let Tomi do the introduction:

If you’ve heard Nokia spinning a story about a Nielsen survey of Lumia owners loving it, and 95% saying they would recommend Lumia to their friends, that sounds very suspicious, doesn’t it?” [2]

YES, Tomi! Yes it does! As suspicious as the Yankee survey you ask us to follow. Any explanation for it?

So whats the story in Balamory? The Nielsen ‘survey’ is a PAID study by Nokia !!! It is utterly biased and non-credible.” [2]

PAID study“? Have you heard the term “there is no such thing as free lunch“? It may be surprise to Tomi, but there are no unpaid surveys either. Somebody needs to pay for the work done, you know?
But seriously, let’s have a look at Nielsen survey, shall we? Tomi did not give reference to either survey, which is always suspecting news. We indeed know Nokia hired Nielsen to do the survey back in April. It’s said in the news – they don’t hide or deny it. So I don’t find it as a breaking news. (And if it was “utterly biased and non-credible”, I would assume Nokia to keep as much in the shadows as possible what comes to their connection to it.)
Naturally Nokia needs to hire some agency to do surveys as they don’t have some secret “survey division” of their own. But biased data as results of such surveys do not serve purpose of the one ordering the survey (they need to know how much their product is liked) or the one performing the survey (they need to keep their credibility).
Question then is: how credible is Nielsen? Looking at their own website is not objective either, so let’s go look at Wikipedia:

Nielsen is a publicly held global information and media company, and is one of the world’s leading suppliers of marketing information (Nielsen Consumer, formerly ACNielsen), media information and TV ratings (Nielsen Media Research), online intelligence (Nielsen Online) and mobile measurement (Nielsen Mobile).” [6]

I’d say it is reasonable to assume that Nielsen does not do biased surveys on demand. But there is one thing here that Tomi doesn’t (intentionally or not) say: survey was done BEFORE it turned out current Lumia’s don’t get full WP8 OS update, but just stripped-down refresh. So results could be lower today, or then not. At least I would like to hear what would the results be if survey was redone.

For Nielsen survey we know Nokia paid for it. That explains why it focuses on Lumia 900. For Yankee Group “genuinely unbiased neutral survey” one can only wonder why they focused on a single model of one manufacturer and what was the motive of Yankee’s client.

Tomi questions Nielsen as unreliable. I have to admit I find Yankee group survey just as questionable. Probably truth lies in between.