When is data statistically significant, and how do you calculate it?

• • • • • •

Learn how to make data-driven decisions with confidence through statistical significance testing.

Let’s say you receive survey results where one answer is selected 40 times and another 60 times, or respondents rate one ad concept a 4.8 in appeal versus another at 4.4. How can you determine if these differences are statistically significant or just due to random chance? This is where the power of statistical significance testing comes into play.

Statistical Significance Calculator

Total Sample A Enter the total sample size for variant A.

Conversions/Successes A Enter the number of conversions/successes for variant A.

Total Sample B Enter the total sample size for variant B.

Conversions/Successes B Enter the number of conversions/successes for variant B.

Hypothesis Choose 'One-sided' to test if variant A is better than variant B. Choose 'Two-sided' to test if there is any difference between the two variants. One-sided Two-sided Confidence Level Select the confidence level for the test: 90%, 95%, or 99%. Higher confidence levels require stronger evidence to reject the null hypothesis. 90% 95% 99%

The results of the statistical significance calculations will be displayed here.

Statistical significance testing is a vital tool in Suzy’s suite of tools, adding rigor to your research and ensuring that each insight you derive is both actionable and accurate. By leveraging Suzy’s advanced analytics, you're equipped to make informed decisions efficiently, optimizing your strategies and outcomes with confidence. Let’s take a look at statistical significance, how to calculate it, and how you can use Suzy’s Data Explorer for fast statistical analysis.

What is statistical significance and why is it important?

In market research, statistical significance is a measure of how likely it is that a result was not caused by chance. In other words, it's a way to determine whether or not a difference between two groups is meaningful. This is an important distinction because, without statistical significance, there’s a chance your results aren’t actionable or require further research to validate. When statistical significance testing is applied to results, it adds a layer of rigor to your research. It’s also common for statistical significance to be referenced during data storytelling and to make a case for a business decision based on your study’s results. 

There are several methods to calculate statistical significance, and there are a few factors that determine which method is best to use, like sample size, the distribution of your dataset, or standard deviation. Two popular methods for calculating statistical significance are Z-tests and T-tests. 

Z-tests require that the standard deviation is known and that you’re working with a higher sample size (over 30 is the industry standard). T-tests, on the other hand, are used when either of those requirements is not met: so either the standard deviation is unknown or you’re working with a lower sample size (below 30 is industry standard).

How statistical significance testing works

There are many methods to calculate statistical significance. The one you choose will depend on a few factors, like sample size or the distribution of your data set. 

At Suzy, statistical significance testing is available when working with a sample size of at least 50 respondents, and we calculate statistical significance by using a Z-test, which is a formula for a hypothesis test.

In a hypothesis test, there are always two hypotheses at play: A null hypothesis, which states that there are not significant or meaningful differences, and an alternate hypothesis, which states that there are significant or meaningful differences. 

Here’s an example: Imagine you’re running a logo test where each logo is a different color, and you want to understand which logo your consumers like the most. 

So the null hypothesis would be that the color does not matter and there are not significant or meaningful differences in performance amongst the different logo colors. The alternative hypothesis would be that the color does matter and there are significant differences in performance amongst the different logo colors. 

With statistical significance testing, we’re trying to prove that the alternative hypothesis is true—that the color of the logo does have a statistically significant impact on logo performance. This evidence comes from a method like a Z-test or T-test in the form of a confidence level.

Confidence level and statistical significance

A confidence level of 95% is the industry standard for results to be considered statistically significant, but depending on the type of research you’re conducting, some brands will accept lower confidence levels. You can interpret a confidence level to mean “we can say with 95% confidence that this difference is not due to chance and is meaningful.” 

Statistical significance testing is automated in the Suzy consumer insights platform for all rating and ranking questions, as well as rating concept questions in monadic surveys at a confidence level of 90% and 95%. With Suzy’s Data Explorer, you can go even deeper in your data analysis by stat testing at a confidence level of 80% and 85%.

Choosing the right confidence level for your brand comes down to finding the right balance of rigor in your study. With too much rigor/too high a confidence level, you may not be able to observe any statistically significant data in your data. If you have too little rigor/too low a confidence level, you’ll have lower statistical power behind anything statistically significant you find in your study. 

Last words

With our advanced analytics, Suzy empowers you to achieve precision in your decisions, giving you the flexibility to target specific markets and manage your data with unmatched control. This precision is crucial when evaluating the significance of your data, allowing you to quickly identify meaningful relationships within your results and differentiate them from those that may require further validation.

See how our Data Explorer and our consumer research platform work, book a demo with our team today!