6 things to know about confidence levels in statistical significance testing
• • • • • •
What is a confidence level and how do you use it in statistical significance testing?
How confident can you be in your market research results?
Statistical significance testing is an essential tool in gathering insights and making data-driven decisions. It helps researchers determine if observed results have occurred by chance or if they are something worth paying attention to. It’s also critical to take sampling errors into account when interpreting results from statistical tests. This is done by calculating confidence levels, which indicate how sure we can be in our results.
But what confidence level should you use when you research? Is it better to use one that is higher? And when can you get away with using a lower confidence level? In this blog, let’s go over confidence level basics, including when to use certain levels and what to use for your research goals.
1. What is a confidence level?
Confidence level is a measure of how confident the researcher is that the results they observe are not due to chance.
Deciding whether your data is statistically significant is usually a researcher’s prerogative. Typically the confidence level is set at 95%, which is the industry standard. This means that when statistical significance is calculated, there is a 95% probability that the observed difference is not due to chance. With that confidence level, researchers can be confident that the results are within 5% of the true population value, which is enough accuracy for many surveys.
2. Statistical power vs. confidence level
Statistical power and confidence level are related concepts, but they represent different aspects of hypothesis testing. Statistical power refers to the probability of correctly rejecting a false null hypothesis, indicating a true effect. It is influenced by factors like sample size, effect size, and significance level.
Confidence level, on the other hand, represents the degree of certainty associated with a statistical estimate, often expressed as a percentage.
As the power of a test increases, the confidence level usually decreases, and vice versa. Researchers must find a suitable balance between power and confidence level based on their study goals.
3. When do researchers use a higher confidence level?
When the stakes are higher, you want a higher confidence level so that you can be more "sure" that what you are seeing is not just random chance. The higher the confidence level, the lower the risk of incorrectly rejecting a null hypothesis and drawing an incorrect conclusion.
4. Can you ever achieve a 100% confidence level?
It’s impossible to attain a confidence level of 100% because there will always be a certain amount of sampling error or bias that could lead to incorrect results. It is also impossible to eliminate the risk of sampling errors or bias. So, unfortunately, you cannot be 100% confident in your results. But you can still get pretty close.
5. When do researchers use a lower confidence level?
In some circumstances, using a lower confidence level is perfectly fine. For example, if there is no significance observed at 95%, the researcher may accept a 90% or even an 85% or 80% confidence level, depending on the situation.
6. How do you choose a confidence level?
Confidence levels are a researcher's choice! Choosing the right confidence level comes down to finding the right balance of rigor for your study. Too much rigor (too high of a confidence level / too discriminating), and you might not observe any statistically significant findings in your dataset. Too little rigor (too low of a confidence level / too relaxed), and there is lower confidence or statistical power behind any statistically significant findings in your dataset.
Statistical significance testing and confidence levels at Suzy
Before delving into the details, it's important to understand that statistical significance testing in the Suzy platform is an automated process at a 90% to 95% confidence level. But in Data Explorer, you can explore 80% to 85% confidence levels.
Data Explorer allows two confidence levels to be used at once. Uppercase letters represent a higher confidence level and lowercase letters represent a lower confidence level.
At Suzy, you have the flexibility to choose the confidence level you need to meet your project goals. To see how statistical testing and confidence levels work at Suzy, book a demo with our team to see our product in action.
