Email A/B Testing: Subject Lines, Send Times, Content & Offers, Sample Sizes, Significance, Holdout Groups & Winner Roll-Out Value
Run email tests you can trust in 20 hands-on labs. You will read a subject line test with open rates, point differences and relative lift, choose the success metric, plan the split, measure the noise of a rate with a confidence interval, test open rates for significance with a z-test and size test cells before sending. Then you find the smallest lift a list can detect on clicks, evaluate send-time, multi-variant content and offer tests, measure incremental orders with holdout groups and size the holdout, value the roll-out of a winner, build a what-if grid for the test share, avoid the peeking problem, break results down by segment, analyse a test log, check whether a lift lasts, prioritise a test roadmap and finish with a yearly scorecard. 20 hands-on labs, fictional stationery shop Quillhaven Paper Co., every cell checked as you go.
- One-time payment
- Instant access right after payment
- Secure checkout with Stripe
About this course
This course teaches email A/B testing as a working method, one spreadsheet lab at a time. In the 20 labs you calculate open rates, point differences and relative lift for subject line tests, pick the right success metric between opens, clicks and orders, plan test cells and the remainder, and put a confidence interval around a rate. You run the two-proportion z-test, work out the sample size per variant and the smallest lift your list can detect, and evaluate send-time, content and offer tests, including the correction for several variants. You measure what an email really adds with holdout groups, value the roll-out of a winner, build a what-if grid for the test share, learn why peeking produces false winners, break a result down by segment, keep a test log and build a yearly scorecard. Every lab uses data from Quillhaven Paper Co., an online stationery shop, and every formula is checked as you type. The company, its subscribers, campaigns and all numbers are fictional, and benchmarks are illustrative.
Why learn with Tabellab
- Practise in a real spreadsheetEvery lab runs in your browser and checks your cells instantly.
- Verifiable certificatePass the final exam and get a certificate with a public verification code.
- Try before you buyLesson 1 is free, with no sign-up or card needed.
- Who is behind TabellabBuilt by Babala Software in Füllinsdorf, Switzerland. Questions? hello@tabellab.com · Imprint
Lessons
- Reading a Subject Line A/B Test: Open Rate per Variant, Point Difference, Relative Lift and the WinnerFree
- Choosing the Success Metric: Click Rate, Click-to-Open Rate, Orders per 1,000 and When Opens Mislead Locked
- Planning the Split: Test Share, Recipients per Variant, the Remainder and a Minimum Cell Size Locked
- How Noisy Is an Open Rate? Standard Error, Margin of Error and a 95% Confidence Interval Locked
- Is the Difference Real? The Two-Proportion z-Test for Open Rates with a p-Value and a Verdict Locked
- Sample Size per Variant: Baseline Open Rate, Minimum Detectable Lift, Confidence and Power Locked
- Testing on Clicks: The Minimum Detectable Lift of a List and How Many Sends a Test Needs Locked
- Send-Time Tests: Click Rate per Slot, Ranking the Slots, Lift over the Usual Time and Results by Time Band Locked
- Content Tests with Several Variants: Each Against the Control, p-Values and the Multiple-Comparison Correction Locked
- Offer Tests: Conversion Rate, Revenue per Recipient and the Margin After the Offer per 1,000 Emails Locked
- Holdout Groups: Incremental Order Rate, Incremental Orders and Revenue, and the Orders That Would Have Come Anyway Locked
- Sizing the Holdout: Margin Lost per Send, Margin of Error of the Lift and the Smallest Holdout That Can Confirm It Locked
- Winner Roll-Out Value: The Remainder, the Effect Inside the Test Cell, the Roll-Out Gain and the Net Value of a Test Locked
- How Large Should the Test Cell Be? A What-If Grid of Test Share against the True Lift Locked
- The Peeking Problem: Results Hour by Hour, Early False Winners and How Long to Wait Locked
- Segment Breakdown: Does the Winner Win Everywhere? Click Rates per Segment and Variant with SUMIFS Locked
- The Test Log: Test Duration from Dates, Significant Wins, Win Rate and Average Winning Lift by Test Type Locked
- Does the Lift Last? Weekly Lift against a Control Group, Its Trend with SLOPE and a Forecast Locked
- The Test Roadmap: Expected Yearly Value of a Test Idea, Net Value after Cost, Ranking and the Top Three Locked
- Final Project: The Testing Programme Scorecard with Win Rate, Net Value and ROI per Quarter, a Flag and a Chart Locked
Learner reviews
Be the first to review this course.