A/B Test Significance Calculator
Analyze your A/B test results.
Load Example Data
Control (A)
Variant (B)
Enter visitor and conversion data for both variations
What Is A/B Test Significance Analysis?
After running an A/B test, you need to determine whether the observed difference is real or just random noise. This calculator analyzes your results:
Inputs:
- Visitors and conversions for each variant
- Significance level (typically 0.05)
- One-tailed or two-tailed test
Outputs:
- P-value
- Statistical significance (yes/no)
- Confidence interval for the difference
- Relative lift
Reading Your Results
P-Value
The probability of seeing a difference this large (or larger) if both variants performed equally.
| P-Value | Interpretation |
|---|---|
| p < 0.01 | Strong evidence against equal performance |
| p < 0.05 | Moderate evidence (conventionally "significant") |
| p < 0.10 | Weak evidence |
| p ≥ 0.10 | Insufficient evidence of difference |
Confidence Interval
The range of plausible true differences:
"Variant B converts 1.2 percentage points higher (95% CI: 0.4 to 2.0)"
If the CI excludes 0, the result is statistically significant at that confidence level.
Relative Lift
The percentage improvement:
Lift = (Rate_B - Rate_A) / Rate_A × 100%
"Variant B shows a 15% relative improvement"
Example Analysis
Test Results:
- Control (A): 10,000 visitors, 320 conversions (3.20%)
- Treatment (B): 10,000 visitors, 385 conversions (3.85%)
Analysis:
- Absolute difference: 0.65 percentage points
- Relative lift: 20.3%
- P-value: 0.032
- 95% CI for difference: [0.06, 1.24] percentage points
Conclusion: B significantly outperforms A (p < 0.05). The true improvement is likely between 0.06 and 1.24 percentage points.
Statistical Significance vs. Practical Significance
Statistically significant: Difference unlikely due to chance alone.
Practically significant: Difference large enough to matter for business.
A 0.01% lift can be statistically significant with millions of visitors but may not justify implementation costs.
Always consider:
- Implementation effort
- Revenue impact
- User experience
- Technical debt
Common Scenarios
Clear Winner
"B converts at 5.2% vs A at 4.0%, p = 0.001"
Strong evidence B is better. Implement B confidently.
No Clear Winner
"B converts at 4.2% vs A at 4.0%, p = 0.35"
No significant difference. Options:
- Continue testing for more data
- Choose based on secondary metrics
- Default to simpler implementation
Surprising Loser
"B converts at 3.5% vs A at 4.0%, p = 0.02"
B is significantly worse. Don't implement B. Investigate why it failed.
One-Tailed vs. Two-Tailed
Two-tailed (recommended): Tests whether variants differ in either direction. Use when B could theoretically be worse than A.
One-tailed: Tests only whether B is better. Lower p-value, but ignores possibility B is worse.
P-value (one-tailed) = P-value (two-tailed) / 2
The Calculation Method
This calculator uses a z-test for two proportions:
z = (p̂_B - p̂_A) / √[p̂(1-p̂)(1/n_A + 1/n_B)]
Where p̂ is the pooled proportion.
For small samples or extreme proportions, exact methods may be more appropriate.
Interpreting Non-Significant Results
"No significant difference" doesn't mean "no difference"—it means you can't rule out chance.
Check the confidence interval:
- CI includes 0 and both meaningful positive and negative effects: Inconclusive, need more data
- CI includes 0 but only small effects: Conclude there's no large difference
Multiple Variants
Testing A vs. B vs. C requires adjustment for multiple comparisons:
Without correction: Each comparison at α = 0.05 Risk of false positive across all comparisons increases.
With Bonferroni correction: Use α / number of comparisons More conservative but controls overall error rate.
Common Mistakes
Early Stopping
Stopping when you first see p < 0.05 inflates false positives. Run to planned sample size.
Ignoring Effect Size
A p-value of 0.04 doesn't tell you if the effect is large enough to matter. Report confidence intervals.
Cherry-Picking Metrics
Testing many metrics and reporting only significant ones inflates false positives. Pre-specify primary metrics.
Ignoring Segments
Overall non-significance might hide significant effects in segments (or vice versa). But segment analysis requires more caution about multiple comparisons.
Power Check
If the test is non-significant, consider:
- Was the test adequately powered?
- What was the minimum detectable effect?
- Did you have enough data to detect a meaningful difference?
Next Steps After Analysis
If B wins convincingly: Implement B, monitor post-launch.
If no clear winner: Consider testing longer, testing bigger changes, or accepting current design.
If B loses: Don't implement B. Learn from why it failed.
Related Calculations
- A/B Test Sample Size Calculator — Plan before testing
- CI for Difference in Proportions — Detailed CI calculation
- Conversion Rate CI Calculator — CI for each rate
- Bayesian A/B Test Calculator — Alternative interpretation
- Minimum Detectable Effect Calculator — What could you detect?
Summary
A/B test significance analysis determines whether observed conversion rate differences are real or random. The p-value quantifies evidence against equal performance, while confidence intervals show the range of plausible effects. Statistical significance indicates a real difference exists; practical significance indicates it matters for business. Report both the p-value and confidence interval, run tests to planned sample size, and always consider whether the observed effect is large enough to justify action.