Confidence Interval shows the range where the true value likely falls. Instead of saying "lift is 10%", a confidence interval says "lift is between 5% and 15% with 95% confidence."
What Confidence Intervals Tell You
A 95% confidence interval means: if you repeated this test many times, 95% of the intervals would contain the true value.
Example:
- Observed : 12%
- 95% CI: [8%, 16%]
This means you're 95% confident the true lift is between 8% and 16%.
Interpreting Confidence Intervals
Narrow interval: High precision. Enough data to estimate the effect accurately.
Wide interval: High uncertainty. Results could vary significantly from the point estimate.
Interval includes zero: The effect might be negative, zero, or positive. Not statistically significant.
Interval entirely above zero: Positive effect is statistically significant.
CI vs Point Estimates
Point estimate: "Conversion increased 10%."
With confidence interval: "Conversion increased 10% (95% CI: 3% to 17%)."
The interval provides crucial context. A 10% lift with CI [9%, 11%] is very different from 10% with CI [-5%, 25%].
Factors Affecting Interval Width
Sample size: More data = narrower intervals.
Variance: Consistent behavior = narrower intervals.
Confidence level: 99% CI is wider than 95% CI.
Effect size: Doesn't affect interval width directly, but smaller effects need more data to measure precisely.
Common Mistakes
-
Ignoring intervals: Focusing only on point estimates misses uncertainty.
-
Misinterpreting width: Wide intervals aren't "wrong" - they honestly reflect limited data.
-
Confusing with probability: A 95% CI doesn't mean 95% probability the true value is in that range. It means 95% of similarly constructed intervals would contain the true value.
