A/B Test Sample Size Calculator

Plan your A/B test sample size.

Please enter a valid baseline rate between 0 and 1

Examples

Enter baseline rate and MDE to calculate sample size

What Is A/B Test Sample Size Calculation?

A/B testing compares two versions (A and B) to see which performs better. Before running the test, you need to know how many users are required to reliably detect a difference.

This calculator determines sample size based on:

  • Baseline conversion rate
  • Minimum detectable effect (lift)
  • Significance level
  • Statistical power

The Key Inputs

Baseline Conversion Rate

Your current performance before the test:

  • "Current checkout completion: 3.2%"
  • "Email open rate: 18%"
  • "Button click rate: 12%"

Minimum Detectable Effect (MDE)

The smallest improvement worth detecting:

Absolute MDE: "Detect a 0.5 percentage point increase" (3.2% → 3.7%)

Relative MDE: "Detect a 15% relative lift" (3.2% → 3.68%)

Smaller MDEs require larger samples.

Significance Level (α)

False positive rate (detecting difference when none exists):

  • α = 0.05 (5%): Standard for most tests
  • α = 0.10 (10%): More lenient, smaller sample needed
  • α = 0.01 (1%): More stringent, larger sample needed

Power (1 - β)

Probability of detecting a real difference:

  • 80%: Standard minimum
  • 90%: Higher confidence, larger sample needed

Sample Size Requirements

Sample size increases when:

  • Baseline rate is closer to 50%
  • MDE is smaller
  • Power is higher
  • Significance level is lower

Example Calculations

Standard e-commerce test:

  • Baseline: 3% conversion
  • MDE: 10% relative lift (to 3.3%)
  • α = 0.05, power = 80%
  • Required: ~35,000 per variant

High-traffic test:

  • Baseline: 20% click rate
  • MDE: 5% relative lift
  • α = 0.05, power = 80%
  • Required: ~12,000 per variant

Low-baseline test:

  • Baseline: 0.5% purchase rate
  • MDE: 20% relative lift
  • α = 0.05, power = 80%
  • Required: ~180,000 per variant

Duration Estimation

Convert sample size to test duration:

Duration = (Required sample per variant × 2) / (Daily visitors × Eligible %)

Example:

  • Need: 25,000 per variant
  • Daily visitors: 10,000
  • Eligible for test: 50%

Duration = 50,000 / 5,000 = 10 days

Common Scenarios

Can't Reach Sample Size

If required sample exceeds available traffic:

  • Accept larger MDE (detect only big improvements)
  • Increase significance level (accept more false positives)
  • Test higher-traffic pages
  • Run test longer
  • Consider not testing (make decision based on judgment)

Very Low Baseline Rates

Purchase rates of 0.1-1% require massive samples. Options:

  • Test higher-funnel metrics (add to cart instead of purchase)
  • Accept larger MDEs
  • Use Bayesian methods for earlier decisions

Multiple Variants

Testing A vs. B vs. C vs. D requires larger samples:

  • More comparisons = more Type I error risk
  • Apply multiple comparison corrections
  • Or: Test sequentially (A vs B, then winner vs C)

One-Tailed vs. Two-Tailed

Two-tailed (recommended): Tests whether B differs from A in either direction. Use this unless you're certain B can't be worse.

One-tailed: Tests only whether B is better than A. Smaller sample, but misses if B is actually worse.

Practical Considerations

Minimum Test Duration

Even with adequate sample, run tests for at least:

  • 1 full week: Capture day-of-week effects
  • 2 full weeks: More robust
  • Full business cycle: Capture seasonal patterns

Visitor vs. Session vs. User

Define your unit of analysis:

  • Visitor: Each session counts independently
  • User: Same user across sessions counts once
  • Conversion rate definition must match sample size calculation

Excluding Bots

Bot traffic inflates sample but not conversions, biasing results. Filter before calculation.

Test Planning Framework

  1. Define metric: What exactly are you measuring?
  2. Determine baseline: What's current performance?
  3. Set MDE: What's the smallest meaningful improvement?
  4. Calculate sample size: How many visitors needed?
  5. Estimate duration: How long will that take?
  6. Go/no-go decision: Is the test feasible?

After the Test

Once you've collected sufficient data:

Common Mistakes

Stopping Too Early

"We hit significance after 2 days!" Early stopping inflates false positive rates. Run to planned sample size.

Peeking Without Correction

Checking results repeatedly increases false positives. Either:

  • Wait until the end
  • Use sequential testing methods designed for continuous monitoring

Ignoring Practical Significance

Statistical significance ≠ business importance. A 0.01% lift might be significant with enough data but worthless in practice.

Related Calculations

Summary

A/B test sample size planning ensures your test can reliably detect meaningful improvements. Based on baseline rate, minimum detectable effect, and desired confidence, the calculator determines visitors needed per variant. Smaller MDEs require larger samples—there's no escaping this tradeoff. Plan before testing, run to completion, and don't stop early just because you see significance. Proper planning prevents wasted tests that can't answer the question.

Related Tools