Published in General

Statistical Significance for the Rest of Us

Suppose you head up a company with one million customers. You’ve heard of this ‘customer experience’ thing, and decide you’ll have a go at it. You design the perfect survey that will tell you exactly how satisfied each respondent is, and you head off to gather your customers’ opinions.

The first month showed that you had an 80% satisfaction level.

The second month showed a great improvement! 90% satisfaction level! w00t! You start handing out raises to everybody, throwing big celebrations!

What’s wrong with this picture? You, dear reader, don’t have enough to go on. What if I told you that each month, you sampled only 10 customers? 0.001% of your customer base. If that were the case, it would be foolish to celebrate a 10% increase. Why?

This is a crude example of what statistical significance is. In our example, the likelihood of either month having an overall satisfaction level of 70% is pretty high, if you surveyed only 10 different customers. There’s a lot of chance involved when you have a sample size that is so small. The 10% increase was not statistically significant. You would likely feel much better if the sample size was 1,000 each month, and showed a 10% increase, wouldn’t you? That would more likely be statistically significant.

So, some really smart mathematicians have come up with formulas that analyze your data and can then tell you if you should get excited or not. These formulas can tell you if a .01% increase in your customer satisfaction average is significant, or if a 5% drop in your Top 2 Box score is something to worry about. They take into account how large (or small) the number of responses is, how spread out your response values are (Standard Deviation), and how certain you want to be that your sample measurement is significant (Alpha, aka Confidence.) When all these things are considered, statistical significance can help ‘even the data out’ when you have a smaller number of respondents. But, if the sample size is too small, there’s no magic here that can fix that.

Note that statistical significance is meant to be done on data that is normally distributed – or at least nearly so. If you are comparing top box scores over time, for instance, and one month, your top box score is 30% based on 200 respondents, but the next month, your top box score is 99% based on 100 respondents, our statistical significance calculator is going to say your sample size is too small, because the second month’s data is so skewed. But, if this were truly the case, you wouldn’t need statistical significance to tell you that your results were probably very good.

A good rule of thumb to keep in mind: The larger the number of respondents, the more likely the results are to be statistically significant.


Acknowledgements: I am deeply indebted to Michelle Pfister, Kyle LaMalfa and Kurt Pflugheoft for their patience and input for this article.