A/B Test Significance Calculator

Why you need it

An A/B test without a significance check is a coin flip with extra steps. A 2% conversion gap on 200 clicks is almost certainly noise; the same gap on 20,000 clicks is real. This calculator runs a two-proportion z-test and returns the p-value — the probability of seeing a difference this large if the variants were actually identical.

The most common mistake in affiliate testing is stopping early. Variant B pulls ahead after 300 visitors, you kill A — and the next thousand clicks flatten the difference completely. Peeking is not illegal, but acting on a lead before significance is how you burn the exact budget the test was meant to save.

Typical uses in media buying: two landing pages on a split test, two angles of the same offer, two prelanders before scaling. The discipline is simple — decide the confidence level before the test (95% is standard), run until you reach it, then make the call. If the result stays insignificant after a proper sample, the variants are equivalent for practical purposes: pick the cheaper one to run.

Worked example: landing A converts 100 of 1,000 visitors (10%), landing B converts 140 of 1,000 (14%). The uplift looks huge — 40% relative — and the calculator confirms it: p ≈ 0.006, significant at 95% and even 99%. But shrink the sample to 100 vs 100 visitors with 10 vs 14 conversions, and the same 40% uplift collapses into insignificance. That is the whole lesson: the effect size and the sample size are inseparable.

For campaigns running in Loza CRM, postback-attributed conversions feed the same math — this tool is for the quick check before you commit a week of spend to a winner that might be noise.