P-Value is the probability of seeing your test results (or more extreme results) if there were no real difference between control and variant. A low p-value suggests the difference is real, not random chance.
What P-Value Means
P-value answers: "If there's no actual difference, how likely is this result?"
P-value = 0.05 means:
- 5% chance this result is random
- 95% confidence the difference is real
- Meets standard significance threshold
P-value = 0.01 means:
- 1% chance this result is random
- 99% confidence the difference is real
- Exceeds standard threshold
Interpreting P-Values
p < 0.05: Statistically significant at 95% confidence. Standard threshold for most tests.
p < 0.01: Highly significant. Strong evidence of real difference.
p > 0.05: Not statistically significant. Cannot rule out random chance.
p > 0.10: Weak evidence. Results likely inconclusive.
What P-Value Is NOT
P-value is often misunderstood:
Not the probability variant is better: P-value doesn't tell you the chance of being right. It tells you the chance of seeing this data if null hypothesis is true.
Not effect size: Low p-value doesn't mean large improvement. A tiny lift can be statistically significant with enough data.
Not certainty: p = 0.05 doesn't mean 95% certainty. It means if you ran 100 tests with no real effect, 5 would show false positives.
P-Value in Practice
Most tools display p-value or its inverse (confidence level):
- "95% confidence" = p-value of 0.05
- "99% confidence" = p-value of 0.01
- "90% confidence" = p-value of 0.10
Common Mistakes
-
P-hacking: Running tests until you get p < 0.05. This inflates false positives.
-
Ignoring practical impact: Statistically significant doesn't mean business significant.
-
Binary thinking: p = 0.049 vs p = 0.051 shouldn't change decisions dramatically. Consider the full picture.
