Tabellab

Reading a Subject Line A/B Test: Open Rate per Variant, Point Difference, Relative Lift and the Winner

Lesson 1 of 20Lab

All lessons
  1. Reading a Subject Line A/B Test: Open Rate per Variant, Point Difference, Relative Lift and the WinnerLabFree
  2. Choosing the Success Metric: Click Rate, Click-to-Open Rate, Orders per 1,000 and When Opens MisleadLab Locked
  3. Planning the Split: Test Share, Recipients per Variant, the Remainder and a Minimum Cell SizeLab Locked
  4. How Noisy Is an Open Rate? Standard Error, Margin of Error and a 95% Confidence IntervalLab Locked
  5. Is the Difference Real? The Two-Proportion z-Test for Open Rates with a p-Value and a VerdictLab Locked
  6. Sample Size per Variant: Baseline Open Rate, Minimum Detectable Lift, Confidence and PowerLab Locked
  7. Testing on Clicks: The Minimum Detectable Lift of a List and How Many Sends a Test NeedsLab Locked
  8. Send-Time Tests: Click Rate per Slot, Ranking the Slots, Lift over the Usual Time and Results by Time BandLab Locked
  9. Content Tests with Several Variants: Each Against the Control, p-Values and the Multiple-Comparison CorrectionLab Locked
  10. Offer Tests: Conversion Rate, Revenue per Recipient and the Margin After the Offer per 1,000 EmailsLab Locked
  11. Holdout Groups: Incremental Order Rate, Incremental Orders and Revenue, and the Orders That Would Have Come AnywayLab Locked
  12. Sizing the Holdout: Margin Lost per Send, Margin of Error of the Lift and the Smallest Holdout That Can Confirm ItLab Locked
  13. Winner Roll-Out Value: The Remainder, the Effect Inside the Test Cell, the Roll-Out Gain and the Net Value of a TestLab Locked
  14. How Large Should the Test Cell Be? A What-If Grid of Test Share against the True LiftLab Locked
  15. The Peeking Problem: Results Hour by Hour, Early False Winners and How Long to WaitLab Locked
  16. Segment Breakdown: Does the Winner Win Everywhere? Click Rates per Segment and Variant with SUMIFSLab Locked
  17. The Test Log: Test Duration from Dates, Significant Wins, Win Rate and Average Winning Lift by Test TypeLab Locked
  18. Does the Lift Last? Weekly Lift against a Control Group, Its Trend with SLOPE and a ForecastLab Locked
  19. The Test Roadmap: Expected Yearly Value of a Test Idea, Net Value after Cost, Ranking and the Top ThreeLab Locked
  20. Final Project: The Testing Programme Scorecard with Win Rate, Net Value and ROI per Quarter, a Flag and a ChartLab Locked

You are not signed in — the lab works, but your progress is not saved. Sign up

Lesson 1 of 20 is free. Unlock all 20 hands-on labs, the final exam and your certificate.

Buy course — $9.90
Loading the spreadsheet…

What a subject line A/B test tells you

The subject line decides whether an email is opened at all. Most teams write two or three candidates, argue about them for a while and then send the one the most senior person likes. An A/B test replaces the argument with a small experiment: two random parts of the list get the same email with different subject lines, and the numbers show which one more people opened.

The method is simple, but reading the result is where many reports go wrong. A dashboard says "B won with 28% against 25%", somebody writes "3% better" in the summary, somebody else reads that as trivial, and a third person quotes it as "12% better" a week later. All three numbers describe the same test. This free lesson shows you how to calculate and name them correctly in Excel: the open rate per variant, the difference in percentage points, the relative lift and the winner.

The example company is Quillhaven Paper Co., a fictional online stationery shop with a weekly newsletter. All campaigns, subscribers and numbers are invented for learning, and any rate you see here is illustrative, not a benchmark. You will work with the same ten tests as in the lab above.

How a subject line test works

A subject line test has four ingredients. First, an audience of subscribers who have agreed to receive the newsletter. Second, a random split: the email tool draws two test cells by chance, so that the cells differ in nothing but luck. Third, one change: the subject line, and only the subject line. Fourth, a comparison on a metric that was chosen before the send, here the open rate.

How a subject line A/B test runsAudienceconsented subscribersTest cell Ausual subject lineTest cell Bchallenger subjectCompareopen ratesRemaindergets thewinnerrandom splitsame send time, same contentafter the waiting time
Figure 1: The audience is split at random, each test cell gets one subject line, and the better one is sent to the remainder.

In practice the two test cells are often only a part of the list, for example 10% each. After a waiting time the tool sends the better subject line to the remaining 80%, the remainder. That way most subscribers receive the stronger version. How large the test cells should be is a topic of its own (lessons 3, 6 and 14 of this course); for reading a finished test, all you need are four counts: delivered and opened emails for A and for B.

A/B testing is done with consented subscribers and changes only how a message is presented. It is not a licence for misleading subject lines: a subject that promises something the email does not contain may win the open and lose the reader's trust, and in many countries deceptive subject lines are also against the law.

The three numbers: open rate, point difference and relative lift

The open rate of a variant is its opens divided by its delivered emails. "Delivered" means sent minus bounced, and each variant has its own count, because a few addresses bounce in one cell and not in the other. In the first Quillhaven test, variant A was delivered to 4,000 subscribers and opened by 1,012 of them, an open rate of 25.30%. Variant B was delivered to 3,988 and opened by 1,128: 28.28%.

The difference in percentage points is the simple subtraction rate B − rate A: 28.28% − 25.30% = 2.98 points. Percentage points are the unit for the gap between two percentages. They answer the question "how many more subscribers out of 100 opened?" Here: about three more out of every hundred.

The relative lift expresses the same gap as a share of the starting point: rate B ÷ rate A − 1 = 28.28% ÷ 25.30% − 1 = 11.8%. It answers "by how much did the open rate grow compared with the usual version?" Variant A is the base because it is the version you would have sent anyway.

Write the unit every time. "B was 3.0 points ahead, a relative lift of 11.8%" cannot be misread. "B was 3% better" can mean either, and the two readings differ by a factor of four.

A fully worked example: ten tests at Quillhaven Paper Co.

Over ten weeks the newsletter team tested a challenger subject line (B) against their usual descriptive style (A). Each variant went to a random test cell of roughly 4,000 subscribers. These are the raw counts, exactly as they appear in the lab sheet in columns A to E:

CampaignDelivered AOpens ADelivered BOpens B
Spring planner launch4,0001,0123,9881,128
Fountain pen guide3,9501,1063,9621,070
Back-to-desk sale4,1201,1954,1051,314
Notebook restock3,8909343,902952
Gift wrap ideas4,0601,1374,0481,052
Journaling challenge3,9751,0733,9901,237
Desk calendar preview4,0108824,022965
Wax seal kits3,9401,0243,9251,029
Loyalty points reminder4,0851,2674,0701,180
Holiday card bundles4,0301,3304,0411,455

Notice that the delivered counts are never quite equal. In the Fountain pen guide test, A reached 3,950 inboxes and B 3,962. That is normal, and it is the reason why you must not compare open counts directly: 1,106 opens against 1,070 looks like a gap of 36 opens, but part of any such gap can come from the cell sizes alone. Rates remove that problem.

Step 1: the open rate of each variant

In the lab, the open rate of variant A goes into column F and that of variant B into column G. The formulas for row 4, copied down to row 13, are:

  • Open rate A (F4): =C4/B4, opens A divided by delivered A.
  • Open rate B (G4): =E4/D4, opens B divided by delivered B.
  • Format both columns as percentages with two decimals. The cell still holds the fraction (0.2530); the format only changes how it is shown.

No dollar signs are needed here: every row has its own four counts, so all references are relative and move down with the formula. Across all ten tests together, variant A was opened by 27.4% of its recipients and variant B by 28.4%. That overall view already suggests that the challenger style is somewhat stronger, but the interesting part is test by test.

Step 2 and 3: point difference and relative lift

Column H holds the difference in points and column I the relative lift:

  • Difference (H4): =G4-F4. If you prefer a formula that does not depend on the helper columns: =E4/D4-C4/B4.
  • Relative lift (I4): =G4/F4-1, or in one step =(E4/D4)/(C4/B4)-1.
  • Both are formatted as percentages. Read column H as points and column I as percent.

Filled down, the sheet gives the following results:

CampaignOpen rate AOpen rate BDifference (points)Relative liftWinner
Spring planner launch25.30%28.28%+2.98+11.8%B
Fountain pen guide28.00%27.01%−0.99−3.5%A
Back-to-desk sale29.00%32.01%+3.00+10.4%B
Notebook restock24.01%24.40%+0.39+1.6%B
Gift wrap ideas28.00%25.99%−2.02−7.2%A
Journaling challenge26.99%31.00%+4.01+14.9%B
Desk calendar preview22.00%23.99%+2.00+9.1%B
Wax seal kits25.99%26.22%+0.23+0.9%B
Loyalty points reminder31.02%28.99%−2.02−6.5%A
Holiday card bundles33.00%36.01%+3.00+9.1%B

Open rate of variant A and variant B, ten testsVariant A (usual)Variant B (challenger)Spring planner launch25.3%28.3%Fountain pen guide28.0%27.0%Back-to-desk sale29.0%32.0%Notebook restock24.0%24.4%Gift wrap ideas28.0%26.0%Journaling challenge27.0%31.0%Desk calendar preview22.0%24.0%Wax seal kits26.0%26.2%Loyalty points reminder31.0%29.0%Holiday card bundles33.0%36.0%
Figure 2: Open rates per variant. B is ahead in 7 of the ten tests, A in 3.

Take the Journaling challenge row. Open rate A is 26.99%, open rate B is 31.00%. The difference is 4.01 points and the relative lift 14.9%, the largest of the ten. In the Gift wrap ideas test the picture is reversed: B's 25.99% is below A's 28.00%, so the difference is −2.0 points and the lift −7.2%. A negative lift simply means the usual subject line did better.

Step 4: naming the winner

The winner column uses one logical test. If B's open rate is higher than A's, the cell shows "B", otherwise "A":

  • Winner (J4): =IF(G4>F4,"B","A"), or without helper columns =IF(E4/D4>C4/B4,"B","A").
  • IF has three parts: the test, the result when the test is true and the result when it is false. Text results go in double quotes.
  • To count the wins, add =COUNTIF(J4:J13,"B") below the table.

In this data set B wins 7 of the ten tests and A wins 3. This is the same rule most email tools apply when they pick a winner automatically after the waiting time: the higher rate wins, however small the gap.

How to read the result

A count of 7 to 3 sounds like a clear verdict for the challenger style. Look at the size of the gaps before you believe it. Figure 3 shows the relative lift of each test.

Relative lift of B over A in the open rate+11.8%Planner−3.5%Pen guide+10.4%Desk sale+1.6%Restock−7.2%Gift wrap+14.9%Journaling+9.1%Calendar+0.9%Wax seals−6.5%Loyalty+9.1%HolidayGreen: B ahead · red: A ahead · bars near zero are too close to call without a significance test
Figure 3: Relative lift per test, from −7.2% (Gift wrap ideas) to +14.9% (Journaling challenge).

Five of B's wins are solid-looking gaps of two to four points (Spring planner launch, Back-to-desk sale, Journaling challenge, Desk calendar preview, Holiday card bundles). The other two are Notebook restock with 0.39 points and Wax seal kits with 0.23 points. On test cells of about 4,000 subscribers, a gap of a few tenths of a point is well within what chance alone produces: send the same subject line to two random cells and you will regularly see differences of that size. Calling those two tests "wins" says more about the IF formula than about the subject lines.

So a more careful reading of the ten tests is: five clear leads for B, two for A (Gift wrap ideas and Loyalty points reminder, about two points each), and three that are too close to call: the two narrow B leads and Fountain pen guide, where A's lead of about one point is also within the range of chance for cells of this size. Whether a gap is large enough to trust depends on the gap and on the number of recipients, and it is answered by a significance test. That is the subject of lesson 5; lesson 4 prepares it by measuring how noisy a single open rate is.

Points versus percent: the classic confusion

Because the open rate is itself a percentage, there are two legitimate ways to describe a change, and they produce very different numbers. For Holiday card bundles the open rate rose from 33.00% to 36.01%. That is +3.00 points and +9.1 percent. Neither is wrong; they answer different questions.

  • Use points when you want to know how many additional people were reached: 3 points on 80,000 subscribers are about 2,400 more opens.
  • Use relative lift when you compare tests with different baselines. Two points on a base of 22% (Desk calendar preview) is a lift of 9.1%; three points on a base of 33% (Holiday card bundles) is 9.1%. The relative view shows that both ideas improved the open rate by about the same proportion.
  • Never add or average relative lifts across tests without thinking: a lift of 10% on a small base and a loss of 10% on a large base do not cancel out.

Common mistakes and how to spot them

  • Dividing by the wrong base. =E4/B4 divides B's opens by A's deliveries. The error is small when the cells are similar and large when they are not. Each variant has its own denominator.
  • Comparing counts instead of rates. =E4-C4 or =IF(E4>C4,"B","A") rewards the bigger cell. Always compare rates.
  • Calling points percent. If your summary says "3% better" check which column the number came from.
  • Reversing the lift. =F4/G4-1 measures A against B. The base of a lift is the version you would have sent without a test, here A.
  • Typing the result instead of a formula. A typed 28.28% does not update when the counts change, for example when late opens arrive.
  • Reading every higher rate as a win. The IF formula has no idea how large a gap must be. Treat gaps of a few tenths of a point as open questions.

What makes the comparison fair

The arithmetic is only meaningful if the test itself was clean. Before trusting any winner label, check five things:

  1. Random split. The cells must be drawn at random by the email tool. Splitting by signup date, alphabet or region compares audiences, not subject lines.
  2. One change. If B also has a different preheader, sender name or send time, you cannot tell which change caused the gap.
  3. Same moment. Both variants go out at the same time. A sent on Tuesday and B on Thursday is a send-time test with a subject line mixed in.
  4. Metric chosen in advance. Decide before the send whether opens, clicks or orders name the winner. Picking the metric afterwards lets you always find one that favours your favourite.
  5. Enough time. Opens keep arriving for a day or two. A verdict after one hour compares fast openers only (lesson 15 shows how that goes wrong).

Why opens are a noisy signal

An open is recorded when the images of an email are loaded. That has two consequences. Readers whose mail app blocks images can read the email without being counted, and mail apps that load images automatically for privacy reasons can record an open that no person saw. The share of such machine opens differs between audiences and changes over time, so open rates from different years or tools are hard to compare.

Inside one A/B test this matters less than you might fear: both cells are random samples of the same list, so both contain a similar share of automatic opens, and the difference between A and B still reflects the subject lines. Still, opens are the weakest evidence in email. For decisions that cost money, confirm a subject line winner on clicks or orders. Lesson 2 of this course compares the click rate, the click-to-open rate and orders per 1,000 emails for exactly that reason.

Step by step in Excel

  1. Put the campaign names in column A and the four counts in columns B to E: delivered A, opens A, delivered B, opens B. Export them from your email tool's test report.
  2. In F4 type =C4/B4, in G4 =E4/D4. Select both cells and press Ctrl + Shift + %, then add two decimals.
  3. In H4 type =G4-F4 and in I4 =G4/F4-1. Format both as percentages.
  4. In J4 type =IF(G4>F4,"B","A").
  5. Select F4:J4 and double-click the fill handle (the small square at the bottom-right corner of the selection) to copy all five formulas down to the last test.
  6. Check one row by hand with a calculator. If the sign of the difference and of the lift disagree, one of the formulas is reversed.
  7. Optional: add =COUNTIF(J4:J13,"B") to count B's wins and a bar chart of column I to see the size of the lifts at a glance.
Keep the raw counts in the sheet, not only the rates. Every later step of a serious analysis, from confidence intervals to significance tests, needs the number of delivered emails. A report that only stores "25.3% vs 28.3%" cannot be checked.

Quick reference: formulas in this lesson

Function / techniqueWhat it does hereExample
DivisionOpen rate of one variant=C4/B4
Subtraction of ratesDifference in percentage points=G4-F4
Ratio minus 1Relative lift of B over A=G4/F4-1
IFWinner label from a comparison=IF(G4>F4,"B","A")
COUNTIFCounts how often B won=COUNTIF(J4:J13,"B")
Percent formatShows 0.2828 as 28.28%Ctrl + Shift + %
Fill handleCopies a formula down the columnDouble-click the corner of the cell

Where this is used in practice

  • Weekly newsletter. Test two subject lines on a part of the list, send the better one to the rest, and record the four counts in a log.
  • Automated emails. Welcome, restock and basket reminder emails go out every day. A subject line test there runs for weeks and the winner keeps paying for as long as the automation runs.
  • Reporting to a client or manager. State open rates, the point difference and the relative lift side by side, and say whether the gap was tested for significance.
  • Building a style guide. After twenty logged tests you can see which kinds of subject lines tend to win for your audience: questions, numbers, product names, deadlines. Lesson 17 turns such a log into win rates by test type.

What you will build in the rest of this course

Lesson 1 is the free preview. Lessons 2 to 20 each have their own live lab with a new data set from Quillhaven Paper Co. and take you from reading a test to running a whole testing programme.

  • Metrics and planning (lessons 2 to 4). You compare click rate, click-to-open rate and orders per 1,000 emails and see a curiosity subject win on opens and lose on orders. You plan test cells and the remainder, and you measure the noise of an open rate with a standard error and a 95% confidence interval.
  • Significance and sample size (lessons 5 to 7). You run the two-proportion z-test with NORM.S.DIST, calculate the recipients per variant needed for a given lift with NORM.S.INV, and find the smallest lift a list can detect in a click test.
  • Other kinds of tests (lessons 8 to 10). You evaluate a send-time test with twelve slots, a content test with seven variants and the correction for multiple comparisons, and an offer test judged on margin after the cost of the offer.
  • Incrementality and value (lessons 11 to 14). You measure what an email adds with a holdout group, size the holdout, put a money value on rolling out a winner and build a what-if grid for the best test share.
  • Running a programme (lessons 15 to 20). You see how peeking at early results produces false winners, break a result down by behavioural segment, analyse a test log, check with SLOPE and FORECAST.LINEAR whether a lift lasts, prioritise a roadmap of test ideas and finish with a yearly scorecard, followed by the exam and the certificate.

Summary

A subject line A/B test produces four counts, and four formulas turn them into a result you can report without ambiguity: =C4/B4 and =E4/D4 for the open rates, =G4-F4 for the difference in percentage points, =G4/F4-1 for the relative lift and =IF(G4>F4,"B","A") for the winner. In the Quillhaven example B was ahead in 7 of ten tests, with lifts across the ten tests between −7.2% and +14.9%, and two of B's wins were too narrow to mean much.

Now scroll up to the lab and build the sheet yourself. It takes about a quarter of an hour, the Coach checks every step, and afterwards you can drop the counts of your own last test into the same five columns.

Frequently asked questions

What is an A/B test of an email subject line?

Two random parts of a mailing list receive the same email with two different subject lines. The open rates of the two groups are compared, and the better subject line is usually sent to the rest of the list.

How do I calculate the open rate of a variant in Excel?

Divide the opens of the variant by its own delivered emails, for example =C4/B4 for variant A and =E4/D4 for variant B, and format the cells as percentages. Each variant has its own denominator.

What is the difference between percentage points and relative lift?

Percentage points are the simple difference between two rates: 28% minus 25% is 3 points. The relative lift divides the new rate by the old one and subtracts 1: 28% divided by 25% minus 1 is 12%. Both describe the same result, so always name the unit.

How do I label the winner of an A/B test with a formula?

Use IF to compare the two rates: =IF(G4>F4,"B","A") returns B when B's open rate is higher and A otherwise. The formula does not check whether the gap is statistically significant; that needs a separate test.

Is the variant with the higher open rate always the real winner?

No. Small gaps, such as a few tenths of a point on a few thousand recipients, are often chance. A significance test such as the two-proportion z-test shows whether the gap is larger than the noise, and clicks or orders show whether more opens also led to more business.

Are open rates still reliable?

Open rates are noisy because opens are counted through image loads, which some mail apps block and others trigger automatically. Within one random A/B test both groups are affected alike, so the comparison still works, but important decisions should be confirmed with clicks or orders.

Keep going: Email A/B Testing: Subject Lines, Send Times, Content & Offers, Sample Sizes, Significance, Holdout Groups & Winner Roll-Out Value

This free lab is lesson 1 of 20. Unlock the other 19 hands-on labs, the final exam and your certificate for a one-time $9.90.

  1. Choosing the Success Metric: Click Rate, Click-to-Open Rate, Orders per 1,000 and When Opens Mislead
  2. Planning the Split: Test Share, Recipients per Variant, the Remainder and a Minimum Cell Size
  3. How Noisy Is an Open Rate? Standard Error, Margin of Error and a 95% Confidence Interval
  4. Is the Difference Real? The Two-Proportion z-Test for Open Rates with a p-Value and a Verdict
  5. Sample Size per Variant: Baseline Open Rate, Minimum Detectable Lift, Confidence and Power
  6. Testing on Clicks: The Minimum Detectable Lift of a List and How Many Sends a Test Needs
  7. Send-Time Tests: Click Rate per Slot, Ranking the Slots, Lift over the Usual Time and Results by Time Band
  8. Content Tests with Several Variants: Each Against the Control, p-Values and the Multiple-Comparison Correction
  9. Offer Tests: Conversion Rate, Revenue per Recipient and the Margin After the Offer per 1,000 Emails
  10. Holdout Groups: Incremental Order Rate, Incremental Orders and Revenue, and the Orders That Would Have Come Anyway
  11. Sizing the Holdout: Margin Lost per Send, Margin of Error of the Lift and the Smallest Holdout That Can Confirm It
  12. Winner Roll-Out Value: The Remainder, the Effect Inside the Test Cell, the Roll-Out Gain and the Net Value of a Test
  13. How Large Should the Test Cell Be? A What-If Grid of Test Share against the True Lift
  14. The Peeking Problem: Results Hour by Hour, Early False Winners and How Long to Wait
  15. Segment Breakdown: Does the Winner Win Everywhere? Click Rates per Segment and Variant with SUMIFS
  16. The Test Log: Test Duration from Dates, Significant Wins, Win Rate and Average Winning Lift by Test Type
  17. Does the Lift Last? Weekly Lift against a Control Group, Its Trend with SLOPE and a Forecast
  18. The Test Roadmap: Expected Yearly Value of a Test Idea, Net Value after Cost, Ranking and the Top Three
  19. Final Project: The Testing Programme Scorecard with Win Rate, Net Value and ROI per Quarter, a Flag and a Chart

Buy course — $9.90 See the full course