Is my A/B test result significant?
Your test has ended and version B converted better. The question now is whether the gap is larger than random noise would produce. The two-proportion z-test answers that with a few lines of arithmetic.
Two-proportion z-test
What the test checks
The null hypothesis says both versions have the same true conversion rate and any gap in your sample is chance. The z-test measures how many standard errors apart the two observed rates are. If that distance would be rare under the null hypothesis, you call the result statistically significant.
The calculation step by step
- Work out each rate: pA = conversions A ÷ visitors A, and the same for B.
- Pool the data: p = (conversions A + conversions B) ÷ (visitors A + visitors B).
- Standard error under the null: SE = √( p(1 − p) × (1/nA + 1/nB) ).
- z = (pB − pA) ÷ SE.
- Two-sided p-value = 2 × (1 − Φ(|z|)), where Φ is the standard normal cumulative distribution.
Worked example. A: 420 conversions from 10,000 visitors. B: 480 from 10,000.
- pA = 4.20%, pB = 4.80%, pooled p = 900 ÷ 20,000 = 4.50%
- SE = √(0.045 × 0.955 × 0.0002) = 0.002932
- z = 0.006 ÷ 0.002932 = 2.047
- p-value = 2 × (1 − 0.97965) = 0.0407
0.0407 is below 0.05, so the result is significant at the 5% level. Figures are rounded to the shown digits; the calculator keeps full precision.
The confidence interval tells you more
A p-value says only whether zero is a plausible difference. The interval shows how big the effect could be. For the interval, use the unpooled standard error:
95% interval = (pB − pA) ± 1.96 × SEdiff
In the example, SEdiff = 0.002931, so the interval runs from 0.03 to 1.17 percentage points. B is probably better, but the true gain could be tiny. That matters if the change costs money to keep.
Same rates, different verdicts
Sample size changes the answer even when the rates are identical:
| Visitors per arm | Rates A / B | z | Two-sided p |
|---|---|---|---|
| 10,000 | 4.2% / 4.8% | 2.047 | 0.0407 |
| 5,000 | 4.2% / 4.8% | 1.447 | 0.1479 |
With half the traffic the same 0.6-point gap is no longer significant. That is why the sample should be planned before launch; the sample size planner shows how.
How the calculator computes Φ
Browsers have no built-in normal distribution, so the widget uses the Abramowitz and Stegun polynomial approximation 26.2.17, which is accurate to better than 0.0000001. Its output was tested against exact values: Φ(1.96) = 0.975002 and Φ(2.576) = 0.995002.
Reading results honestly
- Significant does not mean large. Look at the interval and decide if the low end is still worth it.
- Not significant does not prove no effect. The test may simply have been too small.
- Peeking breaks the maths. A p-value is valid for one look at a pre-planned sample.
- The z-test needs enough events. With very few conversions in either group, use an exact test such as Fisher’s instead.
FAQ
Why pooled for the test but unpooled for the interval?
The test assumes the null is true, so both groups share one rate. The interval estimates the real difference, so each group keeps its own rate.
Is 0.05 a law?
No. It is a convention. Pick your threshold before the test, based on the cost of a wrong decision.