Sample Ratio Mismatch (SRM) occurs when your traffic split doesn't match what you configured. If you set 50/50 but observe 52/48, something is wrong. SRM is a red flag that invalidates test results.
What Causes SRM?
SRM indicates a systematic problem in your test setup:
Technical issues:
- Broken variant code causing redirects
- Caching problems showing one variant more
- Bot traffic affecting one variant differently
- Tracking pixels firing inconsistently
Assignment problems:
- Users switching between variants
- Cookie issues causing reassignment
- Browser extensions blocking variant code
- Redirect latency causing bounces
How to Detect SRM
Compare expected vs actual sample sizes using a chi-squared test:
Expected (50/50 split):
- Control: 10,000 visitors
- Variant: 10,000 visitors
Observed:
- Control: 10,400 visitors
- Variant: 9,600 visitors
A chi-squared test determines if this 52/48 ratio is statistically different from 50/50. If < 0.05, you have SRM.
Why SRM Matters
SRM means your results are unreliable. The traffic imbalance could correlate with other factors:
- Faster-loading variant might have fewer bounces
- Technical errors might affect specific user segments
- One variant might lose mobile users
Any of these could create fake lift or hide real improvements.
What to Do When SRM Occurs
-
Stop the test. Results cannot be trusted.
-
Investigate root cause. Check technical implementation, tracking, and assignment logic.
-
Fix the issue. Resolve whatever caused the imbalance.
-
Restart fresh. Don't continue with tainted data.
Prevention
QA your tests: Verify assignment works correctly before launch.
Monitor early: Check sample ratios within first 24 hours.
Use AA tests: Run control vs control to verify your testing infrastructure.
