Testing on Clicks: The Minimum Detectable Lift of a List and How Many Sends a Test Needs
Lessons in this course
- Reading a Subject Line A/B Test: Open Rate per Variant, Point Difference, Relative Lift and the WinnerFree
- Choosing the Success Metric: Click Rate, Click-to-Open Rate, Orders per 1,000 and When Opens Mislead Locked
- Planning the Split: Test Share, Recipients per Variant, the Remainder and a Minimum Cell Size Locked
- How Noisy Is an Open Rate? Standard Error, Margin of Error and a 95% Confidence Interval Locked
- Is the Difference Real? The Two-Proportion z-Test for Open Rates with a p-Value and a Verdict Locked
- Sample Size per Variant: Baseline Open Rate, Minimum Detectable Lift, Confidence and Power Locked
- Testing on Clicks: The Minimum Detectable Lift of a List and How Many Sends a Test Needs Locked
- Send-Time Tests: Click Rate per Slot, Ranking the Slots, Lift over the Usual Time and Results by Time Band Locked
- Content Tests with Several Variants: Each Against the Control, p-Values and the Multiple-Comparison Correction Locked
- Offer Tests: Conversion Rate, Revenue per Recipient and the Margin After the Offer per 1,000 Emails Locked
- Holdout Groups: Incremental Order Rate, Incremental Orders and Revenue, and the Orders That Would Have Come Anyway Locked
- Sizing the Holdout: Margin Lost per Send, Margin of Error of the Lift and the Smallest Holdout That Can Confirm It Locked
- Winner Roll-Out Value: The Remainder, the Effect Inside the Test Cell, the Roll-Out Gain and the Net Value of a Test Locked
- How Large Should the Test Cell Be? A What-If Grid of Test Share against the True Lift Locked
- The Peeking Problem: Results Hour by Hour, Early False Winners and How Long to Wait Locked
- Segment Breakdown: Does the Winner Win Everywhere? Click Rates per Segment and Variant with SUMIFS Locked
- The Test Log: Test Duration from Dates, Significant Wins, Win Rate and Average Winning Lift by Test Type Locked
- Does the Lift Last? Weekly Lift against a Control Group, Its Trend with SLOPE and a Forecast Locked
- The Test Roadmap: Expected Yearly Value of a Test Idea, Net Value after Cost, Ranking and the Top Three Locked
- Final Project: The Testing Programme Scorecard with Win Rate, Net Value and ROI per Quarter, a Flag and a Chart Locked