Bayesian A/B Test Calculator

What Is Bayesian A/B Testing?

Bayesian A/B testing provides probability statements about which variant is better, rather than just "significant or not."

Frequentist: "Is the difference significant at α = 0.05?" Bayesian: "What's the probability that B beats A?"

The Bayesian approach answers the question most people actually want answered.

Key Outputs

Probability of Being Best

"There's an 94% probability that Variant B has a higher conversion rate."

This directly answers "Which should I choose?"

Expected Lift

"B is expected to convert 0.8 percentage points higher than A."

Combines uncertainty about whether B is better with uncertainty about how much better.

Risk Assessment

"If we choose B but A is actually better, the expected loss is 0.1 percentage points."

Helps quantify the downside of a wrong decision.

How It Works

Prior Distribution

Start with beliefs before seeing data:

• Uninformative prior: No strong beliefs (uniform distribution)
• Informative prior: Based on historical data or domain knowledge

Posterior Distribution

Combine prior with observed data using Bayes' theorem:

Posterior ∝ Likelihood × Prior

For conversion rates, this uses the Beta-Binomial model.

Comparison

Compare posterior distributions:

• Simulate many draws from each variant's posterior
• Count how often B > A

Example Analysis

Observed Data:

• A: 5,000 visitors, 180 conversions (3.60%)
• B: 5,000 visitors, 210 conversions (4.20%)

Bayesian Results:

• P(B > A) = 92.3%
• Expected lift: 0.6 percentage points
• 95% credible interval for difference: [-0.2%, 1.4%]

Interpretation: Strong probability B is better, but some chance A is actually better.

Bayesian vs. Frequentist

Stopping Rules

Frequentist: Must run to predetermined sample size. Early stopping inflates false positives.

Bayesian: Can monitor continuously. Stop when probability exceeds threshold (e.g., P(B > A) > 95%).

Interpretation

Frequentist: "If there's no difference, there's a 3% chance of seeing data this extreme."

Bayesian: "There's a 97% probability B is better."

The Bayesian statement is more intuitive.

Choosing Priors

Uninformative (Non-Informative)

Beta(1, 1) = Uniform(0, 1)

All conversion rates equally likely before data. Lets data speak.

Weakly Informative

Beta(10, 200) for a 5% baseline

Concentrates probability around expected rate while allowing data to dominate.

Informative

Based on historical performance with similar tests.

More data from prior → prior matters more → less data needed from test.

Decision Rules

Probability Threshold

"Choose B if P(B > A) > 95%"

Simple and intuitive.

Expected Value

"Choose the variant with higher expected conversion rate"

May favor B even at P(B > A) = 60% if expected upside is large.

Value of Perfect Information

How much would it be worth to know the true rates?

If low, make a decision; if high, collect more data.

Multiple Variants

Bayesian methods extend naturally:

P(A is best) = 5% P(B is best) = 75% P(C is best) = 20%

Choose B with high confidence, or continue testing B vs. C.

Continuous Monitoring

Bayesian analysis allows valid inference at any point:

Day 3: P(B > A) = 68% → Continue testing Day 7: P(B > A) = 89% → Getting confident Day 10: P(B > A) = 96% → Stop, implement B

No need for fixed sample sizes or correction for multiple looks.

Credible Intervals

Bayesian credible intervals have a direct interpretation:

"There's a 95% probability the true conversion rate is between 3.2% and 4.1%"

This is what people often mistakenly think frequentist CIs mean.

Loss Functions

Beyond "which is better," consider expected loss:

Expected loss from choosing B = P(A > B) × E[A - B | A > B]

If expected loss is small (e.g., 0.05 percentage points), the decision is low-risk.

Historical Context

Thomas Bayes (1701-1761) developed the foundational theorem.

Bayesian methods were controversial for most of the 20th century but gained acceptance with:

• Computational advances (MCMC, Monte Carlo methods)
• Recognition of frequentist limitations in practice
• Tech industry adoption for A/B testing

Companies like Google, Microsoft, and Optimizely have embraced Bayesian A/B testing.

Limitations

Prior Sensitivity

Results depend on prior choice, especially with little data. Sensitivity analysis helps.

Computational Complexity

More complex models require simulation rather than closed-form solutions.

Organizational Understanding

Teams familiar with p-values may need training on probability interpretations.

Related Calculations

• A/B Test Sample Size Calculator — Planning (frequentist)
• A/B Test Significance Calculator — Frequentist analysis
• Conversion Rate CI Calculator — Frequentist CIs
• CI for Difference in Proportions — Compare rates

Summary

Bayesian A/B testing provides intuitive probability statements about which variant is better. Unlike frequentist methods, you can monitor continuously, incorporate prior knowledge, and get direct answers to business questions. The result "P(B > A) = 92%" is easier to understand and act upon than "p = 0.04." While requiring different statistical machinery, Bayesian methods align better with how decisions are actually made—assessing probabilities and expected outcomes, not arbitrary significance thresholds.