A/B Test Calculator
Free tool

A/B Test Calculator: how many visitors do you need, and did B really win?

No statistics degree required. Enter how many people saw each version and how many converted, and get a clear answer in plain English: a winner, a loser, or "not enough data yet".

Free for everyone to use, no sign-up. Nothing you type leaves your browser.

  1. PlanFind out how many visitors your test needs before you launch it.
  2. RunShow version A and version B to visitors at random until you reach that number.
  3. CheckEnter the results to see whether B really won, or whether it is just luck.

"Sure" here means: if A and B were really identical, you would wrongly pick a winner in only 5 tests out of 100 (with 95%). Not sure? Keep the standard.

Use this before you launch. It tells you how many visitors each version needs. Too few visitors and you cannot trust the result, whatever it says.

Out of every 100 visitors, how many buy, sign up or click? "5" means 5 in 100. Not sure? Divide last month's conversions by last month's visitors and multiply by 100.

How much better must B be for you to care? Small improvements need many more visitors to spot, so don't ask for more precision than your business needs.

Gives you an estimated duration. Take last month's visitors and divide by 30.

Advanced settings (most people can skip this)

If B really is better by the amount above, this is the chance the test notices it. Statisticians call it "power".

"Only an improvement" needs fewer visitors but cannot tell you that B is worse.

2 means A versus B. More versions need more visitors in total.

How to run a good A/B test in 5 steps

  1. Change one thing. A new headline, a different button, a shorter form. If you change five things at once you will never know which one worked.
  2. Work out the visitors you need first with the "Before" tab, and write the number down.
  3. Split traffic at random and at the same time: half see A, half see B. Do not compare this month with last month.
  4. Wait. Run the test for at least one or two full weeks and until you reach your number. Resist the urge to peek and stop early.
  5. Check once with the "After" tab, then act on the verdict.

A worked example

Sophie runs an online running-shoe shop. About 5 visitors in 100 buy, and she wants to test a new product page. She would only care about an improvement of about +10% (5% becoming 5.5%).

Step 1. The "Before" tab says she needs about 31,234 visitors per version. With 3,000 visitors a day, that is around 21 days, so she plans three weeks.

Step 2. After three weeks, A had 31,400 visitors and 1,570 buyers (5.0%). B had 31,300 visitors and 1,720 buyers (5.5%).

Step 3. The "After" tab (already filled in with these numbers) says B wins: about 10% better. A gap this big would happen by pure luck only about 5 times in 1,000, and the real improvement is most likely between about +3% and +17%. Sophie rolls out B and keeps watching her sales.

Key ideas explained

Conversion rate
The share of visitors who do what you want (buy, sign up, click). 5 buyers out of 100 visitors is a 5% conversion rate.
A/B test
You show two versions of something (A is usually your current one) to different visitors at the same time, then compare how many convert on each.
Sample size (number of visitors)
How many people must see each version before the comparison means anything. Small groups are noisy; big groups are steady. Flip a coin 10 times and get 7 heads: nothing suspicious. Get 700 heads in 1,000 flips: the coin is clearly not fair.
Improvement (lift)
How much better B is than A. A "+10% improvement" on a 5% conversion rate means 5.5%, not 15%. Points and percentages are different: going from 5% to 5.5% is +0.5 percentage points, or +10%.
"Just luck" and statistical significance
Even two identical pages will show slightly different results, because visitors are different people. A result is called statistically significant when a gap this big would be very unlikely to appear by luck alone. This calculator says it as "a gap this big would happen by luck about N times in 100".
How sure you want to be (confidence level)
Your risk appetite. At 95%, you accept being fooled by luck in about 5 tests out of 100. At 99%, only 1 in 100, but you need more visitors.
Likely range of the real improvement (confidence interval)
What you measured is a best guess. The range shows how far the real improvement could reasonably be from it. If the range includes zero, B could be no better than A, or even worse.
Spotting a real win (power)
If B really is better, this is the chance your test notices it. At 80%, a real win is caught 8 times out of 10.
False winner
Declaring B the winner when it is not actually better. Peeking every day and stopping when B looks good is the most common way to create one.

Methodology (for analysts)

Visitors needed per version for a two-proportion test, where p₁ is the current rate, p₂ = p₁ × (1 + improvement), and p̄ = (p₁ + p₂) / 2:

n = ( z(α)·√(2·p̄·(1−p̄)) + z(β)·√(p₁(1−p₁) + p₂(1−p₂)) )² / (p₂ − p₁)²

z(α) is the critical value for the confidence level (1.645 / 1.960 / 2.576 for 90 / 95 / 99% two-sided) and z(β) the value for the power (0.842 for 80%). The result is rounded up.

Results check: the "luck" figure is the two-sided p-value of a pooled two-proportion z-test. The likely range is a Wald confidence interval on the absolute difference p₂ − p₁ with the unpooled standard error; the relative range divides its bounds by the control rate, which is an approximation. The smallest detectable improvement is the lift the sample could detect at 80% power. Observed power is deliberately not reported: it only restates the p-value.

Assumptions: each visitor is counted once, visitors are randomly assigned, groups are independent, and there are enough conversions and non-conversions (roughly 10 or more) in each group for the normal approximation. The calculator does not correct for several versions, repeated looks at the data or sequential testing.

Frequently asked questions

How many visitors do I need for an A/B test?

It depends on your current conversion rate and on the smallest improvement you care about. A low conversion rate or a small improvement means many more visitors. Enter both in the "Before" tab to get the number needed per version.

What does "95% sure" mean?

It means that if A and B were really identical, you would wrongly pick a winner only about 5 times in 100 tests. It does not mean there is a 95% chance that B is better.

Can I stop my test as soon as B looks like it's winning?

Not safely. Numbers swing a lot in the first days, and stopping the moment they look good makes false winners much more likely. Decide the number of visitors in advance and check the result once you reach it.

My result says "no clear winner". Does that mean A and B are the same?

No. It means the test did not collect enough evidence to tell them apart. Any real difference is probably small, or you need more visitors. The likely range in the results shows how big the difference could still be.

How long should I run an A/B test?

Until each version has reached the number of visitors the calculator gave you, and for at least one or two full weeks so that every day of the week is represented.

Can I test more than two versions?

Yes, but every extra version needs its own set of visitors and increases the chance that luck produces a false winner. Keep it to a few versions and be more sceptical of a lone winner.

Related tools