How to Split Test Subject Lines Like a Newsletter with 2.5 Million Subscribers

By Kelly Forst April 20, 2021

Stuck between two subject line ideas?

It’s impossible to know which will yield the best results without running a split test — commonly called an A/B test. A split test gives you real feedback on which subject line your audience was more drawn to opening.

This is how the most successful marketers optimize their open rates. 

Take Morning Brew, for example. Morning Brew is a popular newsletter with more than 2.5 million subscribers. Their daily newsletter includes business and tech news of the day written for a business-minded millennial audience. 

In August 2020, Morning Brew hit a major milestone — 1,000,000 opens of its daily newsletter.

But how did the newsletter achieve that level of success?

Morning Brew’s secret to 1,000,000 email opens

Morning Brew writer Roby Howell spilled the beans on Twitter on how the popular newsletter tests subject lines for every send. 

Here’s how it works:

1. The Morning Brew team tests 4 subject lines every morning. These subject lines are based on catchy, interesting phrases from the news of the day. 
2. Each of the 4 subject lines is sent to a 3% of their audience at 5 a.m.
3. The winning subject line gets sent to the remaining 88% of their subscribers one hour later. 

This strategy lets you quickly test your subject lines against a small portion of your audience and then send the “winner” to the remaining audience.

It’s a smart way to get more opens on each email.

Want to follow in Morning Brew’s footsteps to set up your own split tests like this? Here’s everything you need to know.

Psst…If you want to keep your split testing simple, try starting with a simple 50/50 split test. This will help you learn the ropes and try split testing before stepping into more advanced testing strategies like this one. 

Related: The Ultimate Guide to AB Testing 

How to split test subject lines like Morning Brew

Testing subject lines like Morning Brew is pretty simple. But there are some steps that need to be taken outside of your email service provider to make sure your results are statistically significant.

That means you need to make sure you have a large enough audience for the first part of your send. If your audience for groups A & B are too small, the results won’t be significant. If the percentage is too large, you could miss out on the opportunity to send the winning subject line to more subscribers.

How to set the breakout percentage

In order to figure out how many subscribers should receive version A and version B of your newsletter, you need to take a look at your historical open rates and the size of your email list.

For the sake of this example, let’s say your email list has 10,000 subscribers, and you historically see 20% open rates. 

Using the Optimizely Calculator, change the “Baseline Conversion Rate” to 20% to account for historical open rate to calculate the ideal size of audience for A & B.

If groups A and B need to have 1,100 subscribers each, that means each group is 11% of your total audience. 

So, in this case, we’re looking at doing an 11%/11%/78% split test.

Related: Your Guide to Writing the World’s Best Email Subject Lines

Determining a subject line winner

Once versions A & B are sent, the next step is to determine the winner. The winner will then get sent to the remaining 78% of your audience. 

To evaluate what would go to the remaining group, look at each emails’ results and evaluate them with a split testing calculator.  

Let’s say, for example, initial results are:

Group A: 1,100 Emails sent, 278 emails opened, 21 bounces

Group B: 1,100 Emails sent, 245 emails opened, 33 bounces

Looking at the numbers, it’s obvious that version A yielded better results than version B. However, you should check to ensure there’s enough data to determine if the results are statistically significant (and not by random chance).

Using the Neil Patel AB Testing Calculator, you can take the results and plug them in. Make sure to subtract the number of bounces from your total emails sent. Roving bounces allows you to measure the true open rate, because if an email bounces, there is no chance of an open. 

Ideally, you want to aim for 90%+ statistical significance. But sometimes you need to move quickly. In that case, aim for 80% or above statistical significance. 

Now, you can be confident that sending version A to the remaining 78% of your audience 

This allows you to maximize the number of opens. In this example, we would yield 234 more opens with the winning email going to the remaining 78% of the subscriber list. 

We came to that conclusion by applying the open rate (or conversion rate in Neil Patel’s calculator above) to the remaining audience.

7800*.26=2028

7800*.23=1794

2028-1794=234

Those 234 opens could make all the difference between additional sales.

How to split test subject lines like Morning Brew with AWeber

Want to set up a subject line split test like Morning Brew for your own email marketing? Follow the steps below and start getting better results with your email marketing

Note: Split testing in AWeber is available to AWeber Lite and Plus users. Upgrade your account to unlock this feature or contact our Customer Solutions team. 

Step #1: Create your emails

The first step to split testing is making sure you have 2 versions of your emails already created in the Message Editor. 

Since this is a subject line split test, all other variables of your email should be exactly the same.

Related: Email Split Tests You Can Set Up in 1 Minute 

Step #2: Set up your split test

To create a split test, hover over the Messages tab and click “Split Tests.” Then click the green “Create” button. 

Next, name your split test. It’s helpful to include descriptive language in the name to differentiate one split test from another. This also helps you quickly identify the takeaway from each split test so you can apply learnings to other emails. 

Then, select a segment of your recipients (if you don’t want to send the emails to your entire list).

And, finally, define your split segments based on the formula used above. And click “Save.”

Now it’s time to select your drafts. Make sure your drafts match up to what you defined in your subject line name.

Then, you can go ahead and schedule your broadcast split test message to segment A and segment B. 

Once you send your emails, you can set a determined period of time to wait in order to determine a “winner” following the steps above. 

You can then send that winning email to segment C.

Get more step-by-step instructions on subject line split testing in AWeber by clicking here.