Email A/B Test Calculator

Most email tests are judged the same way. Two versions go out, someone opens the dashboard a few hours later, one number is bigger than the other, and the bigger one wins. It feels like evidence. Often it is a coin toss with a chart attached.

The trouble is not the arithmetic. The trouble is that every important decision about the test was left until after the data arrived: how many people to send to, what difference would matter, when to stop looking and which number to trust. Made late, each of those choices bends toward whatever the dashboard happens to show.

The cure is old and dull. Decide the size first, read the result once, report a range instead of a verdict, and count something that people actually do.

Deciding the size before sending

Start with the rate you already have. Suppose version A, the email you would send anyway, gets clicks from about 3 percent of the people it reaches. That figure has to come from your own past sends; there is no honest universal number to borrow, because lists, offers and audiences differ too much.

Then name the smallest improvement worth acting on. Not the one you hope for. The one below which you would not bother to change anything. Say it is a fifth more clicks, a relative lift of 20 percent, which takes 3.0 percent to 3.6 percent.

Two more choices complete the plan, and both are about mistakes. The significance level is the risk you accept of calling a difference real when there is none; the statistical handbook kept by the United States National Institute of Standards and Technology calls it the risk of rejecting a hypothesis that is in fact true. Five percent is the usual choice. Power is the chance of noticing the lift if it really exists. Many researchers treat 80 percent as adequate, a convention that accepts four times as much risk of missing a real effect as of inventing one.

With those four numbers, the standard formula for comparing two proportions gives the sample size directly. It balances the spread you would see if the versions were the same against the spread you would see if they differed by exactly your chosen lift. The NIST handbook derives the same kind of formula for a single proportion. For this example, the answer is 13,914 per version, or 27,828 in all.

That number is often a shock. On a list of 50,000, the test alone would use 56 percent of it. The arithmetic is telling you something useful before a single email is sent: a list this size can find a 20 percent lift on a 3 percent click rate, but not a 5 percent one. If the size is out of reach, change the plan, not the reading. Accept a larger lift, accept less power, or run the same comparison across several sends.

The formula is an approximation, built on the normal curve standing in for exact counts. So it is fair to ask whether the promised power is real. The calculator’s checks simulated 200 different plans with exact random counts, 4,000 tests each, and in every plan the share of tests that found the planned lift was within 1.8 points of the target.

What the result can and cannot say

Now the counts come back. Version A was sent to 14,000 people and 420 clicked: 3.00 percent. Version B was sent to 14,000 and 497 clicked: 3.55 percent.

The usual test pools the two groups to estimate one shared rate, as if the versions were the same, and asks how surprising the observed gap would be under that assumption. Here the gap is 2.59 standard errors wide, and a gap that large or larger would turn up by chance about once in a hundred tries. The p-value is 0.0097, below the 5 percent level chosen in advance, so the difference counts as significant. Significant is a narrow word. It says the gap is hard to explain as luck. It does not say the gap is large, and it does not say B is 18 percent better, even though 3.55 is 18 percent more than 3.00.

The interval says more. The calculator gives each rate a Wilson interval, the method the NIST handbook presents for proportions: its worth does not depend strongly on the sample size, and its lower limit can never be negative. A’s rate is plausibly anywhere from 2.73 to 3.30 percent, and B’s from 3.26 to 3.87. The difference between them runs from 0.13 to 0.97 percentage points, and the lift from +4.1% to +34.5%.

That range is the real answer. B is very probably better, but by somewhere between a little and a lot. A forecast that assumes the full 18 percent will often disappoint, and the interval says so before anyone builds a budget on it.

Small tests need extra care. When counts in the two-by-two table are small, below about five by a common rule, the normal approximation behind the z test is poor and an exact test is preferred. The NIST handbook calls Fisher’s exact test an excellent choice for small samples, since it works from the counts themselves. The calculator switches to it when an expected count falls below five, and says so.

One more check costs nothing. If A was meant to get half the sends and got noticeably more, the one-proportion test in the NIST handbook flags the split, and the result should wait until someone explains how recipients were assigned.

Why opens are the wrong thing to count

Open rate is the first number most email reports show, and the weakest one to test on. An open is not observed directly. The sender embeds a tiny remote image in the message, and an open is recorded when something fetches that image.

Plenty of things break that link. Many mail services block images by default, and some readers choose text-only mail, so a person who reads every word may never register an open. A preview pane can fetch the image and count an open even when nobody really reads the message. The fetch itself says nothing about whether anyone read it. Neither error is spread evenly across two versions, especially if they differ in images or length.

Clicks and conversions are recorded when a person does something on purpose. They are rarer, so the test needs more sends, but they measure the thing you care about. If opens are all you have, treat any difference as a hint about how mail programs handled the message, not as proof about readers.

The cost of peeking

The most common way to wreck a well-planned test is to keep checking it. Look after a thousand sends, then five thousand, then ten, and stop the moment the p-value dips under 5 percent. Each look is another chance for noise to cross the line, and once it has crossed, the test is over.

Clinical researchers learned this decades ago. When data are analysed repeatedly as observations accumulate, the probability of a false positive rises, which is why sequential trials adjust the level at each interim look, through boundaries such as Pocock’s or through alpha spending functions.

The calculator measures the damage with its own seeded simulation. It runs 4,000 tests in which both versions share exactly the same 3 percent rate, so any declared winner is false. Read once, at the planned 13,914 per version, 4.7 percent of those tests produced a winner, close to the 5 percent promised. Checked ten times along the way, with a stop at the first significant look, 19.1% did. Five looks gave 14.0 percent and twenty gave 24.7 percent.

Nothing about the email changed between those runs. Only the reading habit did. A rule as simple as “decide the size, then look once” keeps the error rate at the level the plan claimed.

Testing several versions at once raises the same problem from a different side. With three variants each compared against the original at 5 percent, the chance that at least one comparison crosses the line by luck is 1 minus 0.95 cubed, or 14.3%. The plain remedy is Bonferroni’s: divide the level by the number of comparisons. The calculator does that when planning, which raises the sends needed, and applies the slightly sharper Holm adjustment when reading the result.

None of this makes an A/B test clever. It makes it honest about what one send to one list can show. The method cannot explain why people clicked, cannot promise the next campaign will behave the same, and cannot rescue a split that was not random. What it can do is stop a coin toss from being dressed as a finding, and that is most of the job.