The four inputs
- Baseline rate p1: the current conversion rate of the control, from recent data.
- Minimum detectable effect: the smallest lift worth detecting, here 10% relative, so 3.2% becomes 3.52%. Choose it from the business case, not from what you hope to see.
- Significance level α: the false-alarm rate you accept, usually 5% two-sided.
- Power 1 − β: the chance of detecting the MDE if it is real, usually 80%.
Relative versus absolute MDE is the most common mix-up: a 10% relative lift on 3.2% is 0.32 percentage points, while a 10-point absolute lift would be 13.2%, a completely different test.
The sample size formula
For two conversion rates, the visitors needed per variant are:
n = (zα/2 + zβ)² × (p1(1 − p1) + p2(1 − p2)) ÷ (p2 − p1)²
In Excel, NORM.S.INV(1-α/2) gives zα/2 = 1.96 for α = 5%, and NORM.S.INV(power) gives zβ = 0.84 for 80% power. Always round up with ROUNDUP(…,0): you cannot run 0.7 of a visitor, and rounding down leaves you slightly under-powered.
Worked example: sample size and duration for a conversion test
The six yellow inputs are all you change. The target rate, the z values, the visitors per variant and the test length are formulas.
| A | B | |
|---|---|---|
| 1 | Input | Value |
| 2 | Baseline conversion rate p1 | 3.2% |
| 3 | Minimum detectable effect (relative lift) | 10% |
| 4 | Significance level α (two-sided) | 5% |
| 5 | Power (1 − β) | 80% |
| 6 | Variants incl. control | 2 |
| 7 | Visitors per day entering the test | 6,500 |
| 8 | ||
| 9 | Target rate p2 = p1 × (1 + MDE) | 3.52% |
| 10 | z for α/2 | 1.9600 |
| 11 | z for power | 0.8416 |
| 12 | Visitors per variant | 49,774 |
| 13 | Visitors in total | 99,548 |
| 14 | Days needed | 16 |
| 15 | Test length in full weeks | 3 |
- B9
=B2*(1+B3)Target rate after the relative lift - B10
=NORM.S.INV(1-B4/2)Critical value for a two-sided test - B11
=NORM.S.INV(B5)z value for the power - B12
=ROUNDUP((B10+B11)^2*(B2*(1-B2)+B9*(1-B9))/(B9-B2)^2,0)Visitors per variant - B14
=ROUNDUP(B13/B7,0)Days needed - B15
=ROUNDUP(B14/7,0)Rounded up to full weeks
Normal approximation for two proportions, two-sided test, equal traffic split. Teaching numbers for a fictional online shop.
From visitors to test duration
Multiply the visitors per variant by the number of variants and divide by the daily traffic that enters the test: 49,774 × 2 = 99,548 visitors ÷ 6,500 a day = 15.3, so 16 days. Then round up to full weeks, here three, because weekdays and weekends convert differently and a test that covers Monday twice but Saturday once is biased.
Decide the duration before you start and do not stop when the result first looks significant. Checking every day and stopping at the first p < 0.05 (peeking) can push the real false-positive rate well above 5%. If you must look early, use a sequential method with stricter boundaries.
Why the MDE drives everything
The effect size sits squared in the denominator, so halving the MDE roughly quadruples the sample. With the same 3.2% baseline:
- 20% relative lift: 13,012 visitors per variant
- 10% relative lift: 49,774
- 5% relative lift: 194,527
A lower baseline needs more traffic too: at 1.6% instead of 3.2%, a 10% lift needs 101,282 per variant. If the numbers are out of reach, test a bolder change, pick a metric with a higher baseline (for example add-to-cart instead of purchase), or accept that the test can only detect larger effects.
More than two variants and more power
Every extra variant compared with the control is another chance of a false winner. A simple fix is the Bonferroni correction: divide α by the number of comparisons. With three challengers, α = 5% ÷ 3, zα/2 rises to 2.39 and the sample grows to 66,390 per variant, for four variants in total.
Raising power from 80% to 90% costs about a third more traffic (66,633 per variant in the example). Online calculators use the same normal approximation, but some use a pooled variance or a continuity correction, so expect results that differ by a few percent.
Common mistakes
- Using an absolute MDE where a relative one is meant. A 10% lift on 3.2% is 3.52%, not 13.2%.
- Reading n as the total. The formula gives visitors per variant. Multiply by the number of variants for the traffic you need.
- Rounding down. Always round the sample size and the duration up.
- Stopping early when it looks significant. Peeking inflates false positives. Fix the duration in advance.
- Running partial weeks. Weekday effects bias the result. Run full weeks.
- Testing many variants at α = 5%. Correct for multiple comparisons or the chance of a false winner grows with every variant.
Frequently asked questions
How do you calculate the sample size for an A/B test?
Use n = (z for α/2 + z for power)² × (p1(1 − p1) + p2(1 − p2)) ÷ (p2 − p1)², where p1 is the baseline rate and p2 the rate after the minimum detectable effect. In Excel the z values come from NORM.S.INV.
What is a good minimum detectable effect?
The smallest lift that would change your decision, given what the change costs. Many teams use 5% to 20% relative. Smaller MDEs need far more traffic: halving the MDE roughly quadruples the sample.
How long should an A/B test run?
Until it reaches the planned sample size, rounded up to full weeks so every weekday is covered equally. Decide this before the test starts and do not stop at the first significant result.
Why do online sample size calculators give different numbers?
They use slightly different variance terms (pooled or unpooled), one- or two-sided tests, or a continuity correction. The differences are usually a few percent.