SpunkArt
Numbers and money maths

How to plan an A/B test sample size

An A/B test only answers your question if enough people see each version. The sample size depends on four choices you make before launch. This page shows the standard formula, a worked example and a planner you can run in your browser.

Sample size planner

The four inputs you choose

  1. Baseline rate (p1). The conversion rate of the current version, taken from your own recent data.
  2. Minimum detectable effect. The smallest change worth acting on. A 20% relative lift on a 4% baseline means you want to detect a move to 4.8%.
  3. Significance level (α). The false-alarm rate you accept. A 95% confidence level means α = 0.05, split across both tails.
  4. Power (1 − β). The chance the test detects the lift if it is real. 80% is a common default; 90% needs more traffic.

The standard two-proportion formula

For two groups of equal size, the sample needed in each group is:

n = [ zα/2 √(2 p̄ (1 − p̄)) + zβ √(p1(1 − p1) + p2(1 − p2)) ]² ÷ (p2 − p1)²

Here p̄ is the average of p1 and p2. The z values come from the standard normal distribution: 1.960 for 95% two-sided confidence, 1.645 for 90%, 2.576 for 99%, 0.842 for 80% power and 1.282 for 90% power. Always round n up to the next whole person.

Worked example. Baseline 4%, target 4.8% (a 20% relative lift), 95% confidence, 80% power.

  • p̄ = (0.04 + 0.048) ÷ 2 = 0.044
  • First term: 1.960 × √(2 × 0.044 × 0.956)
  • Second term: 0.842 × √(0.04 × 0.96 + 0.048 × 0.952)
  • Squared sum divided by 0.008² gives about 10,316.2, so n = 10,317 per variant, or 20,634 in total.

At 1,500 users per day across both versions, that is 20,634 ÷ 1,500 = 13.8 days. Round up to two full weeks so every weekday appears equally in both groups.

How the effect size drives everything

The difference p2 − p1 sits squared in the denominator. Halve the lift you want to detect and you need roughly four times the traffic. All figures below use 95% confidence and 80% power, rounded up.

BaselineRelative liftTarget ratePer variant
4%10%4.4%39,475
4%20%4.8%10,317
4%30%5.2%4,783
4%50%6.0%1,863
2%20%2.4%21,109
10%20%12.0%3,841

Low baseline rates need bigger samples. Raising power from 80% to 90% on the 4% / 20% case moves the requirement from 10,317 to 13,810 per variant.

Planning mistakes to avoid

  • Choosing the lift after you see the data. Fix the minimum effect, confidence and power before launch and write them down.
  • Stopping when the result looks good. Checking every day and stopping at the first good-looking number inflates false positives. Run to the planned sample.
  • Mixing up relative and absolute lift. A move from 4% to 4.8% is 20% relative but only 0.8 percentage points absolute.
  • Splitting traffic across many variants. Each extra version needs its own full sample, and multiple comparisons need a stricter threshold.

When the test has finished, the analysis is a different calculation. The significance test page covers the z-test, the p-value and the confidence interval.

FAQ

Can I use a smaller sample if I only care whether B is better?

Yes. A one-sided test needs fewer users because α is not split. Decide this before the test starts; switching after you see the data is not valid.

What if my traffic is too low for the sample?

Test bigger changes that could plausibly move the rate more, measure a more frequent event higher in the funnel, or accept a longer test. Lowering power just raises the chance of missing a real effect.

Does the planner store my numbers?

No. It runs in your browser and sends nothing anywhere.