How to plan an A/B test sample size
An A/B test only answers your question if enough people see each version. The sample size depends on four choices you make before launch. This page shows the standard formula, a worked example and a planner you can run in your browser.
Sample size planner
The four inputs you choose
- Baseline rate (p1). The conversion rate of the current version, taken from your own recent data.
- Minimum detectable effect. The smallest change worth acting on. A 20% relative lift on a 4% baseline means you want to detect a move to 4.8%.
- Significance level (α). The false-alarm rate you accept. A 95% confidence level means α = 0.05, split across both tails.
- Power (1 − β). The chance the test detects the lift if it is real. 80% is a common default; 90% needs more traffic.
The standard two-proportion formula
For two groups of equal size, the sample needed in each group is:
Here p̄ is the average of p1 and p2. The z values come from the standard normal distribution: 1.960 for 95% two-sided confidence, 1.645 for 90%, 2.576 for 99%, 0.842 for 80% power and 1.282 for 90% power. Always round n up to the next whole person.
Worked example. Baseline 4%, target 4.8% (a 20% relative lift), 95% confidence, 80% power.
- p̄ = (0.04 + 0.048) ÷ 2 = 0.044
- First term: 1.960 × √(2 × 0.044 × 0.956)
- Second term: 0.842 × √(0.04 × 0.96 + 0.048 × 0.952)
- Squared sum divided by 0.008² gives about 10,316.2, so n = 10,317 per variant, or 20,634 in total.
At 1,500 users per day across both versions, that is 20,634 ÷ 1,500 = 13.8 days. Round up to two full weeks so every weekday appears equally in both groups.
How the effect size drives everything
The difference p2 − p1 sits squared in the denominator. Halve the lift you want to detect and you need roughly four times the traffic. All figures below use 95% confidence and 80% power, rounded up.
| Baseline | Relative lift | Target rate | Per variant |
|---|---|---|---|
| 4% | 10% | 4.4% | 39,475 |
| 4% | 20% | 4.8% | 10,317 |
| 4% | 30% | 5.2% | 4,783 |
| 4% | 50% | 6.0% | 1,863 |
| 2% | 20% | 2.4% | 21,109 |
| 10% | 20% | 12.0% | 3,841 |
Low baseline rates need bigger samples. Raising power from 80% to 90% on the 4% / 20% case moves the requirement from 10,317 to 13,810 per variant.
Planning mistakes to avoid
- Choosing the lift after you see the data. Fix the minimum effect, confidence and power before launch and write them down.
- Stopping when the result looks good. Checking every day and stopping at the first good-looking number inflates false positives. Run to the planned sample.
- Mixing up relative and absolute lift. A move from 4% to 4.8% is 20% relative but only 0.8 percentage points absolute.
- Splitting traffic across many variants. Each extra version needs its own full sample, and multiple comparisons need a stricter threshold.
When the test has finished, the analysis is a different calculation. The significance test page covers the z-test, the p-value and the confidence interval.
FAQ
Can I use a smaller sample if I only care whether B is better?
Yes. A one-sided test needs fewer users because α is not split. Decide this before the test starts; switching after you see the data is not valid.
What if my traffic is too low for the sample?
Test bigger changes that could plausibly move the rate more, measure a more frequent event higher in the funnel, or accept a longer test. Lowering power just raises the chance of missing a real effect.
Does the planner store my numbers?
No. It runs in your browser and sends nothing anywhere.