Tabellab
Marketing Analytics Intermediate

Email A/B Testing: Subject Lines, Send Times, Content & Offers, Sample Sizes, Significance, Holdout Groups & Winner Roll-Out Value

Run email tests you can trust in 20 hands-on labs. You will read a subject line test with open rates, point differences and relative lift, choose the success metric, plan the split, measure the noise of a rate with a confidence interval, test open rates for significance with a z-test and size test cells before sending. Then you find the smallest lift a list can detect on clicks, evaluate send-time, multi-variant content and offer tests, measure incremental orders with holdout groups and size the holdout, value the roll-out of a winner, build a what-if grid for the test share, avoid the peeking problem, break results down by segment, analyse a test log, check whether a lift lasts, prioritise a test roadmap and finish with a yearly scorecard. 20 hands-on labs, fictional stationery shop Quillhaven Paper Co., every cell checked as you go.

20 lessons 20 labs Certificate

  • One-time payment
  • Instant access right after payment
  • Secure checkout with Stripe

About this course

This course teaches email A/B testing as a working method, one spreadsheet lab at a time. In the 20 labs you calculate open rates, point differences and relative lift for subject line tests, pick the right success metric between opens, clicks and orders, plan test cells and the remainder, and put a confidence interval around a rate. You run the two-proportion z-test, work out the sample size per variant and the smallest lift your list can detect, and evaluate send-time, content and offer tests, including the correction for several variants. You measure what an email really adds with holdout groups, value the roll-out of a winner, build a what-if grid for the test share, learn why peeking produces false winners, break a result down by segment, keep a test log and build a yearly scorecard. Every lab uses data from Quillhaven Paper Co., an online stationery shop, and every formula is checked as you type. The company, its subscribers, campaigns and all numbers are fictional, and benchmarks are illustrative.

Why learn with Tabellab

  • Practise in a real spreadsheetEvery lab runs in your browser and checks your cells instantly.
  • Verifiable certificatePass the final exam and get a certificate with a public verification code.
  • Try before you buyLesson 1 is free, with no sign-up or card needed.
  • Who is behind TabellabBuilt by Babala Software in Füllinsdorf, Switzerland. Questions? hello@tabellab.com · Imprint

Lessons

  1. Reading a Subject Line A/B Test: Open Rate per Variant, Point Difference, Relative Lift and the WinnerLabFree
  2. Choosing the Success Metric: Click Rate, Click-to-Open Rate, Orders per 1,000 and When Opens MisleadLab Locked
  3. Planning the Split: Test Share, Recipients per Variant, the Remainder and a Minimum Cell SizeLab Locked
  4. How Noisy Is an Open Rate? Standard Error, Margin of Error and a 95% Confidence IntervalLab Locked
  5. Is the Difference Real? The Two-Proportion z-Test for Open Rates with a p-Value and a VerdictLab Locked
  6. Sample Size per Variant: Baseline Open Rate, Minimum Detectable Lift, Confidence and PowerLab Locked
  7. Testing on Clicks: The Minimum Detectable Lift of a List and How Many Sends a Test NeedsLab Locked
  8. Send-Time Tests: Click Rate per Slot, Ranking the Slots, Lift over the Usual Time and Results by Time BandLab Locked
  9. Content Tests with Several Variants: Each Against the Control, p-Values and the Multiple-Comparison CorrectionLab Locked
  10. Offer Tests: Conversion Rate, Revenue per Recipient and the Margin After the Offer per 1,000 EmailsLab Locked
  11. Holdout Groups: Incremental Order Rate, Incremental Orders and Revenue, and the Orders That Would Have Come AnywayLab Locked
  12. Sizing the Holdout: Margin Lost per Send, Margin of Error of the Lift and the Smallest Holdout That Can Confirm ItLab Locked
  13. Winner Roll-Out Value: The Remainder, the Effect Inside the Test Cell, the Roll-Out Gain and the Net Value of a TestLab Locked
  14. How Large Should the Test Cell Be? A What-If Grid of Test Share against the True LiftLab Locked
  15. The Peeking Problem: Results Hour by Hour, Early False Winners and How Long to WaitLab Locked
  16. Segment Breakdown: Does the Winner Win Everywhere? Click Rates per Segment and Variant with SUMIFSLab Locked
  17. The Test Log: Test Duration from Dates, Significant Wins, Win Rate and Average Winning Lift by Test TypeLab Locked
  18. Does the Lift Last? Weekly Lift against a Control Group, Its Trend with SLOPE and a ForecastLab Locked
  19. The Test Roadmap: Expected Yearly Value of a Test Idea, Net Value after Cost, Ranking and the Top ThreeLab Locked
  20. Final Project: The Testing Programme Scorecard with Win Rate, Net Value and ROI per Quarter, a Flag and a ChartLab Locked

Learner reviews

Be the first to review this course.

Write a review

Your rating

Reviews are checked before they appear. By submitting you confirm the images are yours and may be shown on this page. Privacy