When conducting a subject line A/B test, we recommend that your A and B lists each have at least 1,000 contacts for a total of 2,000 contacts. The larger your audience the better because it decreases the chances that your results are an anomaly, and increases your confidence in your outcome; this is called "statistical significance".
Having a high statistical significance means you're more confident in your outcome and can make a better decision based on it, while having a low statistical significance means that your outcome can be due to other factors than you didn't intend.
You're A/B testing your subject line, and send out two versions of your subject line, each to half your list. One of your contacts is checking their email and they get up for a minute to refill their coffee. While they're away, their cat steps on the keyboard and accidentally opens your email, which happens to be subject line B. How important is the cat to your A/B Test results?
| A list of 10 contacts | A list of 100 contacts | A list of 1,000 contacts | A list of 10,000 contacts | |
| A/B test list size for Email B | 5 | 50 | 500 | 5,000 |
| Number of opens (assuming a 25% open rate) | 1 | 12 | 125 | 1250 |
| The cat is responsible for: | 100% of your opens! | 8% of your opens! | .8% of your opens! | .08% of your opens! |
| Statistical Significance | Lowest | Low | High | Highest |
As you can see, the larger your list, the greater the number of opens, and the less the cat matters. Don't forget that to finish your A/B test, you still have to compare the open rate of Email B to Email A - if your list is small, the cat could cause you to declare the wrong winner!
In this example, you know the cat is a factor and therefore you know its impact on your open rate, but you'll never know for sure why an email is opened or unopened when you send to a real contact list. You can't prevent mistaken opens, but you can minimize their impact on your test results by testing with a larger list size.
 | The data shows: Your subject line has a direct impact on your open rate; when your subject line is written well, it peaks the curiosity of your contacts and they open your email to see more detail, and when your subject line is poor, your contacts skip over it. The winning subject line from a subject line A/B test gives you an idea of what your audience is looking for when they skim through their inbox. The next time you send an email to the same audience using your winning subject line as a guide, you should see an increase in opens. |
Copyright © 2025 · All Rights Reserved · Constant Contact · Privacy Center