Email A/B Test Sample Size: How Many Subscribers You Need for a Reliable Result

Most email A/B tests are declared winners too early or on audiences far too small to tell luck from effect. The result is a "winning" subject line that does nothing when rolled out. The fix is not complicated: plan the sample size first, then read the result with the right test. This guide gives you the numbers and the method. For an overview of the whole process, read our email A/B testing guide first.

The four inputs that determine sample size

Input	What it means	Typical choice
Baseline rate	Your current click (or conversion) rate	From past campaigns
Minimum detectable effect (MDE)	The smallest relative lift worth acting on	10–20% relative
Confidence level	How rarely you accept a false positive	95% (α = 5%)
Power	How often you detect a real effect of that size	80% (sometimes 90%)

Smaller baselines and smaller effects need far larger audiences. Halving the lift you want to detect roughly quadruples the sample.

Sample size per variant: reference numbers

Using the standard two-proportion formula at 95% confidence (two-sided) and 80% power:

Baseline	Relative lift to detect	Recipients per variant
2%	+50%	≈ 3,826
5%	+20%	≈ 8,158
20%	+10%	≈ 6,510

These match the calculator below. If your list is smaller than the number you need, you cannot reliably detect that effect: either test something with a bigger expected impact, test on a metric with a higher base rate (opens are higher than clicks but unreliable; see below) or accumulate results across several sends.

⚖️A/B Test CalculatorStatistical significance for two variants with p-value, confidence interval and sample size planningOpen the tool →

The significance test, in plain language

For two rates the usual approach is a two-proportion z-test. It asks: if both versions truly performed the same, how surprising is a gap this large? That probability is the p-value.

• p < 0.05 means a gap this large would occur by chance less than 5% of the time if there were no real difference.
• It is not the probability that B is better, and it says nothing about how big or valuable the difference is.
• "Not significant" is not "no difference". It often means the test was too small. Look at the confidence interval of the difference.

Which metric to judge by

Opens are unreliable because of Apple Mail Privacy Protection, so judge subject-line tests by clicks and design or offer tests by conversions. A higher open rate with no change in clicks usually means a curiosity-bait subject that did not deliver value. Metric formulas are covered in the email metrics formulas guide.

How long to run an email test

1. Randomise and send at the same time so both versions face the same conditions.
2. Wait for the response window to close. Most email engagement arrives within 24–48 hours; waiting at least that long avoids favouring versions sent to quick responders.
3. Collect the planned sample, not "until it looks good". Decide the stopping rule in advance.

Choosing the minimum detectable effect honestly

The most common planning error is picking a tiny effect, such as +2%, because any gain sounds good. Ask instead what lift would change a decision. If a 5% to 5.1% change would not justify rewriting your template, there is no reason to buy the audience needed to detect it. A useful habit is to compute the value of the smallest lift: extra clicks per send multiplied by your conversion rate and average order value. If that number is below the cost of the change, pick a larger effect or skip the test. Write the MDE into the test plan so the result cannot quietly redefine success afterwards.

Segments and new subscribers

Results average across everyone you sent to. A subject line can win with engaged subscribers and lose with those who joined last week, and a pooled number hides that. Randomise within the whole audience, and only afterwards look at one or two pre-planned segments, such as new versus existing subscribers. Treat unplanned segment analysis as a source of new hypotheses, not proof, since slicing enough ways will always produce an apparent winner.

The mistakes that create false winners

Mistake	Why it misleads	Fix
Peeking	Checking repeatedly and stopping at the first p < 0.05 inflates false positives	Fix the sample size first and analyse once
Testing many variants	Each extra comparison adds chance of a lucky result	Limit variants or apply a correction
Changing several things	You cannot tell which change worked	One variable per test
Uneven or non-random splits	Different audiences, not different content, drive the gap	Randomise the whole list
Declaring victory on opens	Opens are inflated and bot-driven	Use clicks or conversions
Over-generalising	A winner for one segment may not hold for another	Re-test important changes

Worked example: planning and reading one test

Your newsletter has a 5% click rate and a list of 40,000. You want to test a new call-to-action and only care about a lift of 20% or more (5% to 6%).

1. Plan: the table above says about 8,158 recipients per variant, so 16,316 in total. The 40,000 list is enough, and you can send the winner to the remaining audience.
2. Send: randomly split 8,200 per variant at the same time, wait 48 hours.
3. Read: A has 410 clicks (5.0%), B has 492 (6.0%). Pooled rate = 902 ÷ 16,400 = 5.5%; z ≈ 2.8, so p ≈ 0.005.
4. Decide: significant, and the interval for the difference (about +0.3 to +1.7 points) stays above zero. Ship B.

Had B produced 440 clicks (5.4%), z would be about 1.1 and p about 0.3: no evidence of a difference at this size, even though B "looks" better. That is the situation peeking tempts people to misread.

When your list is small

NIST recommends the Fisher exact test for small samples, where the normal approximation behind the z-test becomes unreliable. As a practical rule, if you expect fewer than about 10 clicks in any variant, treat a result as a hint and repeat the test. Small lists are better served by testing bold changes (a different offer, not a different comma) and by pooling the same hypothesis across several sends.

A simple decision rule

1. Plan the audience for the smallest lift worth shipping.
2. Send once to the planned sample, randomised.
3. Calculate the p-value and the interval for the difference.
4. If significant and the interval's low end is still worth the change, ship the winner.
5. If not significant, record it as "no evidence of a difference at this size" and move on to a bolder test.

Frequently asked questions

How many people do I need for an email A/B test?

It depends on your baseline rate and the lift you want to detect. For example, detecting a 20% relative lift over a 5% baseline at 95% confidence and 80% power needs roughly 8,100 recipients per variant.

How long should I run an email A/B test?

Send both versions at the same time and wait at least 24 to 48 hours for engagement to settle, then analyse once, after the planned sample has been reached.

Is a 95% confidence level necessary?

It is the common standard for decisions that are costly to reverse. For low-risk changes some teams accept 90%, but do not change the threshold after seeing the data.

What if my list is too small to reach the sample size?

Test bigger changes (larger expected lift), pool several sends of the same hypothesis or accept that you can only detect large effects.

Can I trust the winner from my email platform automatically?

Check the rule it uses. Many tools pick a winner on opens after a short period with small samples, which is often unreliable. Verify with a significance test on clicks.

References & further reading

1. NIST/SEMATECH e-Handbook — 7.3.3 Do two processes produce the same proportion of defectives?
2. Apple — Protect Mail activity (Mail Privacy Protection)
3. NIST/SEMATECH e-Handbook — 7.2.4.1 Confidence intervals