Experiment Power

Experiment Power (or statistical power) is the probability your test will detect a real difference when one actually exists. Low power means you might miss real improvements. High power means you're likely to find real effects.

Understanding Power

Power answers: "If variant B really is 10% better, what's the chance my test will detect it?"

80% power means:

• 80% chance of detecting a real effect
• 20% chance of missing it (false negative)
• Industry standard minimum

90% power means:

• 90% chance of detecting a real effect
• More data required than 80% power
• Used for important decisions

Factors That Affect Power

Sample size: More visitors = higher power. This is the main lever you control.

Effect size: Larger differences are easier to detect. A 20% lift needs less data than a 2% lift.

Baseline conversion rate: Lower rates need more samples to detect changes.

Significance threshold: Stricter thresholds (99% vs 95%) reduce power.

Variance: More consistent user behavior = higher power.

Power Analysis

Run power analysis before testing to determine required sample size:

Inputs:

• Baseline conversion rate (e.g., 3%)
• Minimum detectable effect (e.g., 10% relative lift)
• Desired power (e.g., 80%)
• Significance level (e.g., 95%)

Output:

• Required sample size per variant
• Expected test duration

Why Power Matters

Underpowered tests waste resources:

False negatives: Real improvements go undetected. You reject winners.

Inconclusive results: Tests end without actionable findings.

Wasted traffic: Visitors in failed tests could have been in powered tests.

Common Mistakes

1. Skipping power analysis: Starting tests without knowing required sample size.

2. Unrealistic effect size: Expecting 50% lifts when 5% is realistic.

3. Stopping underpowered tests: Declaring "no winner" when you simply lacked data to detect one.