A/B Testing for Political Texting: A Data-Driven Framework

Gut feelings don't win elections. Data does.

Every campaign believes they know what messages will resonate. But voters constantly surprise us. The message you're certain will crush it flops. The variation you almost didn't test becomes your best performer.

A/B testing removes guesswork. It lets voters tell you what works - then you do more of it.

What Is A/B Testing?

Definition: Sending two (or more) variations of a message to different segments of your audience, measuring performance, and determining which performs better.

Simple example:

Version A: "Hi Sarah! Will you vote on Nov 5?"

Version B: "Hi Sarah! Can we count on your vote Nov 5?"

Send A to 50% of your list, B to 50%, measure response rates, and use the winner for future campaigns.

Why A/B Testing Matters

Small Changes, Big Impact

Even minor message variations can dramatically affect performance:

Real example:

Version A: "Hi Tom! Donate $50 to help us win."

• Response rate: 3.2%

Version B: "Hi Tom! Your $50 helps us reach 200 voters."

• Response rate: 5.8%

Result: 81% increase in response rate from one small change.

In a campaign that sends 500,000 messages, that difference means:

• Version A: 16,000 responses
• Version B: 29,000 responses
• Difference: 13,000 additional responses

Compounding Benefits

A/B testing creates continuous improvement:

Month 1: Test opening lines, find 20% improvement Month 2: Test CTAs on winning version, find 15% improvement Month 3: Test timing, find 10% improvement

Cumulative impact: 50%+ improvement over baseline

What to Test

1. Message Opening

The first words determine whether voters keep reading.

Test variations:

Formal vs. conversational:

• A: "Good morning, Sarah."
• B: "Hey Sarah!"

Personalized vs. generic:

• A: "Hi Sarah!"
• B: "Hi there!"

Question vs. statement:

• A: "Have you voted yet?"
• B: "Polls are open!"

With gratitude vs. without:

• A: "Hi Sarah! Thanks for your support."
• B: "Hi Sarah!"

2. Message Length

Shorter isn't always better.

Test variations:

Ultra-short (under 100 characters):

Hi Tom! Will you vote Nov 5?

Short (100-160):

Hi Tom! Election Day is Nov 5. Your polling place: Lincoln Elementary. Will you vote?

Medium (160-250):

Hi Tom! This is Mike with Johnson for Congress. Election Day is Nov 5 and your vote matters. Your polling place is Lincoln Elementary, open 7 AM-8 PM. Can we count on you?

Measure: Response rate, engagement quality

3. Call-to-Action

Your CTA determines what action voters take.

Test variations:

Direct ask:

• A: "Will you vote?"
• B: "Reply YES if you'll vote"
• C: "Can we count on your vote?"

Specific vs. general:

• A: "Donate now"
• B: "Donate $50 to reach 200 voters"

Urgency levels:

• A: "Vote on Tuesday"
• B: "Don't forget to vote Tuesday"
• C: "Polls close at 8 PM Tuesday - vote now!"

4. Tone and Voice

Test variations:

Urgent vs. calm:

• A: "URGENT: Vote today!"
• B: "Polls are open today - make your voice heard"

Emotional vs. factual:

• A: "This election determines our future"
• B: "Election Day is Nov 5"

Positive vs. negative:

• A: "Vote to protect healthcare"
• B: "Stop them from cutting healthcare - vote!"

5. Personalization Level

Test variations:

Name only:

Hi Sarah! Vote on Nov 5.

Name + location:

Hi Sarah! Vote at Lincoln Elementary on Nov 5.

Name + location + past behavior:

Hi Sarah! Thanks for voting in 2020. Vote at Lincoln Elementary Nov 5!

Name + location + issue:

Hi Sarah! As a teacher, you know education funding is on the ballot Nov 5. Vote at Lincoln Elementary!

6. Timing

Test variations:

Time of day:

• A: Send at 9 AM
• B: Send at 6 PM

Day of week:

• A: Send Tuesday
• B: Send Saturday

Days before event:

• A: Send 3 days before
• B: Send 1 day before

7. Sender Identification

Test variations:

First name only:

• A: "This is Mike with Johnson for Congress"
• B: "This is Mike"

Title inclusion:

• A: "This is Mike, Field Director for Johnson"
• B: "This is Mike with Johnson for Congress"

Candidate name:

• A: "This is Mike with Johnson for Congress"
• B: "This is Mike with Emily Johnson's campaign"

How to Structure A/B Tests

Step 1: Formulate Hypothesis

Don't test randomly. Have a theory.

• ❌ Bad: "Let's test two messages and see what happens"

• ✅ Good: "Hypothesis: Messages with specific polling place information will have higher response rates because they reduce friction"

Step 2: Identify One Variable

Change only one thing at a time.

❌ Bad:

• Version A: "Hi Sarah! Will you vote Nov 5?"
• Version B: "Hey there! Can we count on your support on Election Day?"

(Changed: greeting, formality, CTA, date format)

• ✅ Good:
  • Version A: "Hi Sarah! Will you vote Nov 5?"
  • Version B: "Hi Sarah! Can we count on your vote Nov 5?"

(Changed: Only the CTA)

Step 3: Determine Sample Size

Minimum for statistical significance:

• 1,000+ messages per variation
• At least 50 responses per variation

Better:

• 5,000+ messages per variation
• 100+ responses per variation

For high-volume campaigns:

• 10,000+ messages per variation
• 500+ responses per variation

Step 4: Randomize Assignment

Ensure fair test:

• Randomly split your audience 50/50
• Don't cherry-pick who gets which version
• Ensure segments are comparable

How to randomize:

• Use platform's A/B split feature
• Manually: Sort list randomly, send A to first half, B to second half

Step 5: Set Success Metrics

Before testing, define what "winning" means:

For GOTV:

• Primary metric: Response rate
• Secondary: Positive confirmations

For fundraising:

• Primary metric: Conversion rate (donations)
• Secondary: Average donation amount

For events:

• Primary metric: RSVP rate
• Secondary: Actual attendance

Step 6: Run Test Simultaneously

Send both versions at the same time.

• ❌ Bad: Send version A on Monday, version B on Wednesday

• ✅ Good: Send both on Monday at 2 PM

Why: Time of day/week affects performance. Simultaneous sending isolates the variable you're testing.

Step 7: Collect Data

Track all relevant metrics:

• Messages sent
• Messages delivered
• Responses received
• Response rate
• Opt-outs
• Conversions (if applicable)

Step 8: Analyze Results

Determine statistical significance:

Quick rule: Winner needs at least 10% better performance

Example:

• Version A: 20% response rate
• Version B: 22% response rate
• Difference: 10% - Declare B the winner

More rigorous: Use statistical significance calculators

• Requires larger sample sizes
• Accounts for random variation
• Typical threshold: 95% confidence

Step 9: Implement Winner

Use the winning version for:

• Remainder of current campaign
• Future similar campaigns
• Base for next round of testing

Document learning:

• What won
• By how much
• Why you think it won
• How to apply insight

Advanced A/B Testing

Multivariate Testing

Test multiple variables simultaneously:

Example:

Pros: Faster than sequential A/B tests

Cons: Requires much larger sample sizes (2,000+ per variation)

Sequential Testing

Build on winners:

Round 1: Test opening lines → Winner: "Hi [Name]!"

Round 2: Test CTAs on winning opening → Winner: "Can we count on you?"

Round 3: Test timing for winning message → Winner: 6 PM send

Result: Highly optimized message built step by step

Segment-Specific Testing

Test variations within segments:

Example: Test different messages for young voters vs. seniors

Young voters:

• A: "Hey Alex! Vote to shape climate policy"
• B: "Hi Alex! This election determines climate action"

Seniors:

• A: "Hello Mr. Johnson. Election Day is Nov 5"
• B: "Hi Mr. Johnson! Protect Social Security - vote Nov 5"

Insight: Different audiences may respond to different approaches

Holdout Groups

Reserve a control group:

Setup:

• 40%: Version A
• 40%: Version B
• 20%: No message (control)

Insight: Measure lift from messaging vs. no contact

Common A/B Testing Mistakes

1. Testing Too Many Variables

❌ Changing opening, CTA, length, and tone all at once

✅ Change one variable at a time

2. Insufficient Sample Size

❌ Testing with 100 messages per variation

✅ Use 1,000+ per variation minimum

3. Not Running Simultaneously

❌ Version A on Monday, B on Friday

✅ Both at same time

4. Declaring Winners Too Early

❌ "Version A has 3 more responses after 50 sends - it wins!"

✅ Wait for statistical significance

5. Ignoring Context

❌ "Version A always wins, use it everywhere"

✅ Context matters (audience, timing, campaign phase)

6. Testing Without Hypotheses

❌ Random testing

✅ Test based on theories and insights

7. Not Documenting Results

❌ Test, implement winner, forget

✅ Document learnings for institutional knowledge

A/B Testing Calendar

Early Campaign

Focus: Foundational elements

• Message tone
• Sender identification
• Basic personalization

Frequency: 1-2 tests per month

Mid-Campaign

Focus: Optimization

• CTAs
• Timing
• Segmentation approaches

Frequency: 2-3 tests per month

Final Weeks

Focus: High-impact refinements

• GOTV message variations
• Urgency levels
• Specific polling information

Frequency: 1 test per week (move fast)

Real-World Examples

Example 1: GOTV Message

Hypothesis: Including specific polling hours increases response

Version A:

Hi Sarah! Vote on Nov 5 at Lincoln Elementary.

Version B:

Hi Sarah! Vote Nov 5 at Lincoln Elementary, open 7 AM-8 PM.

Results:

• Version A: 18% response rate
• Version B: 24% response rate
• Winner: B (33% improvement)

Insight: Specific logistics reduce friction

---

Example 2: Fundraising Ask

Hypothesis: Specific impact increases donations

Version A:

Hi Tom! Donate $50 to help us win.

Version B:

Hi Tom! Your $50 funds voter outreach to 200 people.

Results:

• Version A: 2.8% conversion rate
• Version B: 4.2% conversion rate
• Winner: B (50% improvement)

Insight: Concrete impact motivates giving

---

Example 3: Timing

Hypothesis: Evening messages outperform morning

Version A: Send at 9 AM Version B: Send at 6 PM

Results:

• Version A: 16% response rate
• Version B: 23% response rate
• Winner: B (44% improvement)

Insight: Voters more responsive after work

Tools and Platforms

What you need:

A/B testing features:

• Split audience functionality
• Simultaneous sending
• Performance tracking

Analytics:

• Response rates by variation
• Conversion tracking
• Statistical significance indicators

Documentation:

• Test logs
• Results tracking
• Insight repository

Political Comms provides built-in A/B testing with automatic splitting, real-time results, and comprehensive analytics.

The Bottom Line

A/B testing transforms campaigns from guessing to knowing:

Benefits:

• 20-50%+ performance improvements
• Data-driven decisions
• Continuous optimization
• Institutional knowledge

Best practices:

• Test one variable at a time
• Use sufficient sample sizes (1,000+ per variation)
• Run tests simultaneously
• Define success metrics upfront
• Document and apply learnings

What to test:

1. Message opening
2. Call-to-action
3. Tone and urgency
4. Personalization level
5. Timing
6. Message length

Remember: Every campaign, audience, and context is different. Test your assumptions. Let voters tell you what works.

At Political Comms, we make A/B testing easy with built-in tools, automatic splits, and real-time performance tracking.

---

Ready to start testing and optimizing? Get started with Political Comms.

Need help designing tests? Contact our team for expert guidance.