Tabellab

Winner Roll-Out Value: The Remainder, the Effect Inside the Test Cell, the Roll-Out Gain and the Net Value of a Test

Lesson 13 of 20 Lab 🔒 Locked

Lessons in this course

  1. Reading a Subject Line A/B Test: Open Rate per Variant, Point Difference, Relative Lift and the WinnerLabFree
  2. Choosing the Success Metric: Click Rate, Click-to-Open Rate, Orders per 1,000 and When Opens MisleadLab Locked
  3. Planning the Split: Test Share, Recipients per Variant, the Remainder and a Minimum Cell SizeLab Locked
  4. How Noisy Is an Open Rate? Standard Error, Margin of Error and a 95% Confidence IntervalLab Locked
  5. Is the Difference Real? The Two-Proportion z-Test for Open Rates with a p-Value and a VerdictLab Locked
  6. Sample Size per Variant: Baseline Open Rate, Minimum Detectable Lift, Confidence and PowerLab Locked
  7. Testing on Clicks: The Minimum Detectable Lift of a List and How Many Sends a Test NeedsLab Locked
  8. Send-Time Tests: Click Rate per Slot, Ranking the Slots, Lift over the Usual Time and Results by Time BandLab Locked
  9. Content Tests with Several Variants: Each Against the Control, p-Values and the Multiple-Comparison CorrectionLab Locked
  10. Offer Tests: Conversion Rate, Revenue per Recipient and the Margin After the Offer per 1,000 EmailsLab Locked
  11. Holdout Groups: Incremental Order Rate, Incremental Orders and Revenue, and the Orders That Would Have Come AnywayLab Locked
  12. Sizing the Holdout: Margin Lost per Send, Margin of Error of the Lift and the Smallest Holdout That Can Confirm ItLab Locked
  13. Winner Roll-Out Value: The Remainder, the Effect Inside the Test Cell, the Roll-Out Gain and the Net Value of a TestLab Locked
  14. How Large Should the Test Cell Be? A What-If Grid of Test Share against the True LiftLab Locked
  15. The Peeking Problem: Results Hour by Hour, Early False Winners and How Long to WaitLab Locked
  16. Segment Breakdown: Does the Winner Win Everywhere? Click Rates per Segment and Variant with SUMIFSLab Locked
  17. The Test Log: Test Duration from Dates, Significant Wins, Win Rate and Average Winning Lift by Test TypeLab Locked
  18. Does the Lift Last? Weekly Lift against a Control Group, Its Trend with SLOPE and a ForecastLab Locked
  19. The Test Roadmap: Expected Yearly Value of a Test Idea, Net Value after Cost, Ranking and the Top ThreeLab Locked
  20. Final Project: The Testing Programme Scorecard with Win Rate, Net Value and ROI per Quarter, a Flag and a ChartLab Locked