Email A/B Test Plan

When to Use This Skill

Use this skill when you need to:

• Design a structured A/B test plan for email campaigns
• Define clear hypotheses, test variables, and success criteria
• Calculate sample sizes needed for statistically meaningful results
• Build a testing roadmap that compounds email performance over time

DO NOT use this skill for website A/B testing, ad creative testing, or general analytics setup. This is for email-specific split testing.

Core Principle

TEST ONE VARIABLE AT A TIME WITH A CLEAR HYPOTHESIS — A TEST WITHOUT A PREDICTION IS JUST RANDOM VARIATION.

Phase 1: Testing Brief

Required Inputs

GATE: Confirm inputs before designing the test plan.

Phase 2: Test Design

Hypothesis Framework

Every test starts with a hypothesis in this format:

IF we [change this variable]
THEN [this metric] will [increase/decrease]
BECAUSE [reasoning based on audience behavior]

Example:

IF we change the subject line from benefit-focused to curiosity-based
THEN open rate will increase by 5+ percentage points
BECAUSE our audience responds to information gaps more than stated benefits

Testable Variables (One at a Time)

Subject Line Tests:

• Length (short vs. long)
• Formula (curiosity vs. benefit vs. urgency)
• Personalization ({first_name} vs. no name)
• Emoji vs. no emoji
• Number vs. no number

Send Time Tests:

• Morning vs. afternoon
• Weekday vs. weekend
• Specific day comparison (Tues vs. Thurs)

Content Tests:

• Long vs. short email body
• Plain text vs. designed HTML
• Single CTA vs. multiple links
• Image-heavy vs. text-only
• PS line vs. no PS line

CTA Tests:

• Button vs. text link
• CTA copy variations
• CTA placement (top vs. bottom vs. both)
• Color/design of CTA button

Sample Size Guidance

## Minimum Sample Sizes

To detect a 2 percentage point lift at 95% confidence:
- Open rate test: ~2,500 per variant (5,000 total)
- Click rate test: ~5,000 per variant (10,000 total)

For smaller lists:
- Under 1,000: Test is directional only, not statistically significant
- 1,000-5,000: Can test subject lines (open rate) only
- 5,000-10,000: Can test most variables
- 10,000+: Can test subtle differences

GATE: Present the test design with hypothesis for approval.

Phase 3: Build the Test

Test Plan Document

## A/B Test Plan

**Test name:** [Descriptive name]
**Hypothesis:** IF... THEN... BECAUSE...
**Variable tested:** [One specific element]
**Control (A):** [Current version]
**Variant (B):** [Changed version]
**Metric:** [Primary: open rate | Secondary: click rate]
**Sample size:** [X per variant]
**Test duration:** [Send date or date range]
**Winner criteria:** Variant wins if primary metric is [X]% higher with 95% confidence
**Rollout plan:** Winner sends to remaining [X]% of list [X hours] after test

Writing Both Versions

Write the complete Control (A) and Variant (B) versions of whatever is being tested. If testing subject lines, write both. If testing email body, write both complete emails.

Phase 4: Polish

1. Testing Roadmap

Map out 6-8 tests in priority order:

## 12-Week Testing Roadmap

Week 1-2: Subject line formula (curiosity vs. benefit)
Week 3-4: Send time (Tuesday 10am vs. Thursday 10am)
Week 5-6: Email length (200 words vs. 500 words)
Week 7-8: CTA style (button vs. text link)
Week 9-10: Personalization (first name vs. no name)
Week 11-12: Design (HTML template vs. plain text)

2. Results Tracking Template

| Test | Date | Variable | Control | Variant | Lift | Significant? | Action |
|------|------|----------|---------|---------|------|--------------|--------|
| #1 | [Date] | Subject formula | 22% open | 27% open | +5pp | Yes | Adopt curiosity formula |

3. Common Pitfalls Checklist

• Only one variable is different between A and B
• Both versions sent at the same time
• Sample is randomly split (not segmented by behavior)
• Test runs long enough for reliable data (minimum 24 hours for open rate)
• Winner criteria defined before sending

Anti-Patterns

• Testing multiple variables at once — if you change the subject line AND the send time, you cannot attribute results to either change.
• Declaring a winner too early — wait at least 24 hours for open rate tests, 48 hours for click rate tests.
• Testing with tiny samples — 50 people per variant tells you nothing. You need thousands for meaningful results.
• No hypothesis — "Let's just try something different" produces random results you cannot learn from.
• Never implementing winners — testing is pointless if you do not apply what you learn to future emails.

Recovery

• List too small for statistical significance: Run tests as directional indicators. Track patterns across multiple tests instead of relying on single results.
• No baseline metrics: Send 3-4 emails without testing first to establish baseline open and click rates.
• User wants to test 5 things at once: Prioritize by expected impact. Subject lines affect open rates most — start there.
• Platform doesn't support A/B testing: Manually split the list into two segments and send different versions. Compare results in a spreadsheet.