Statistical Significance
A determination that an observed result is unlikely to have occurred by random chance alone. Conventionally indicated by a p-value below 0.05, meaning less than 5% probability of the result being a fluke.
Definition: A determination that an observed result is unlikely to have occurred by random chance alone. Conventionally indicated by a p-value below 0.05, meaning less than 5% probability of the result being a fluke.
Statistical significance is a determination that an observed result—like "Design B got 15% more clicks than Design A"—is unlikely to have occurred by random chance alone.
How It Works
When you measure a sample, there is always a chance that your findings are just due to random noise or the specific people you happened to recruit. A statistical test calculates the probability of observing your result if there were truly no real difference.
If this probability (the p-value) is very low—conventionally below 0.05 (5%)—the finding is called "statistically significant," and you can be more confident that the effect is real.
What It Means
A statistically significant result suggests:
- The difference you observed is probably not a random fluke
- You can be more confident generalizing the finding to the broader population
- The effect is likely to replicate if you ran the study again
What It Does Not Mean
Statistical significance does not tell you:
- How big the effect is (that is effect size)
- Whether it matters practically (a tiny but significant difference may not be worth acting on)
- That the null hypothesis is false (it is about probability, not proof)
Always report effect size alongside significance to show whether a difference is large enough to justify action.
Related Terms
P-Value
The probability of observing your data (or something more extreme) if there were truly no effect. Widely used, widely misunderstood, and never sufficient on its own to make a decision.
Effect Size
A measure of the magnitude of a finding—how big the difference is between conditions, not just whether it exists. Essential for determining practical significance beyond statistical significance.
Quantitative Research
Research focused on numerical measurement with the goal of generalizing findings from a sample to a broader population. Answers 'how much,' 'how many,' and 'how often.'
Mentions in the Knowledge Hub
This term is referenced in the following articles: