When is data statistically significant, and how do you calculate it?

• • • • • •

How do you calculate statistical significance? Learn more about statistical significance testing and how it applies to your Suzy data!

Let’s say you get results back on a survey question and respondents chose one answer 40 times and another answer 60 times, or perhaps respondents rated one of your ad concepts as a 4.8 in appeal and rated another ad concept as a 4.4. How can you tell if those differences are meaningful and not just a result of random chance? Statistical significance testing takes the guesswork out of drawing conclusions from your results, so you can quickly identify which relationships within your results are meaningful and which ones may need further validation.

Statistical significance testing is a powerful tool that can be applied to research data to provide insight into which differences are significant and meaningful.

What is statistical significance and why is it important?

In market research, statistical significance is a measure of how likely it is that a result was not caused by chance. In other words, it's a way to determine whether or not a difference between two groups is meaningful. This is an important distinction because, without statistical significance, there’s a chance your results aren’t actionable or require further research to validate. When statistical significance testing is applied to results, it adds a layer of rigor to your research. It’s also common for statistical significance to be referenced during data storytelling and to make a case for a business decision based on your study’s results. 

There are several methods to calculate statistical significance, and there are a few factors that determine which method is best to use, like sample size, the distribution of your dataset, or standard deviation. Two popular methods for calculating statistical significance are Z-tests and T-tests. 

Z-tests require that the standard deviation is known and that you’re working with a higher sample size (over 30 is the industry standard). T-tests, on the other hand, are used when either of those requirements is not met: so either the standard deviation is unknown or you’re working with a lower sample size (below 30 is industry standard).

How statistical significance testing works

There are many methods to calculate statistical significance. The one you choose will depend on a few factors, like sample size or the distribution of your data set. 

At Suzy, statistical significance testing is available when working with a sample size of at least 50 respondents, and we calculate statistical significance by using a Z-test, which is a formula for a hypothesis test.

In a hypothesis test, there are always two hypotheses at play: A null hypothesis, which states that there are not significant or meaningful differences, and an alternate hypothesis, which states that there are significant or meaningful differences. 

Here’s an example: Imagine you’re running a logo test where each logo is a different color, and you want to understand which logo your consumers like the most. 

So the null hypothesis would be that the color does not matter and there are not significant or meaningful differences in performance amongst the different logo colors. The alternative hypothesis would be that the color does matter and there are significant differences in performance amongst the different logo colors. 

With statistical significance testing, we’re trying to prove that the alternative hypothesis is true—that the color of the logo does have a statistically significant impact on logo performance. This evidence comes from a method like a Z-test or T-test in the form of a confidence level.

Confidence level and statistical significance

A confidence level of 95% is the industry standard for results to be considered statistically significant, but depending on the type of research you’re conducting, some brands will accept lower confidence levels. You can interpret a confidence level to mean “we can say with 95% confidence that this difference is not due to chance and is meaningful.” 

Statistical significance testing is automated in the Suzy consumer insights platform for all rating and ranking questions, as well as rating concept questions in monadic surveys at a confidence level of 90% and 95%. With Suzy’s Data Explorer, you can go even deeper in your data analysis by stat testing at a confidence level of 80% and 85%.

Choosing the right confidence level for your brand comes down to finding the right balance of rigor in your study. With too much rigor/too high a confidence level, you may not be able to observe any statistically significant data in your data. If you have too little rigor/too low a confidence level, you’ll have lower statistical power behind anything statistically significant you find in your study. 

Last words

Statistical significance is a method used to validate the meaningfulness of your results in market research, and it can play a critical role in determining which results are actionable and which ones may need further validation. 

If you're looking for help with your market research, Suzy’s team of in-house market research experts is here to assist you. Book a demo with Suzy today and see how we can help you get started.