Email marketing is both an art and a science. The art comes in with elegant designs, punchy subject lines and well-written copy. Then there’s the science. The data and statistics prove which parts of the art gets people to interact with your email… and which parts don’t work as well. That’s where A/B testing comes into play. Running head-to-head tests allows you to experiment with the creative and understand the effect. Read on to understand the rules of A/B testing, the parts of an email you can test and the metrics you should look at afterwards.
WHAT ARE THE RULES OF A/B TESTING?
Here’s the golden rule of A/B testing: only test one thing at a time. If you change more than one variable, you have no way of knowing what made the difference. When you test a single variable against a control, you can accurately measure the impact it had.
Another important thing to keep in mind: run your tests simultaneously. This goes back to that golden rule. Time-based factors can muddy up the results. It’s another variable, after all. So if you send the control on Monday and then send out the test on Tuesday… you don’t know if the day of the week influenced the results. It’s another variable. (But hey, that’s another test you can run — as long as the emails are exactly the same).
Last but not least: make sure your sample size is large enough. How large is large enough? A good rule of thumb is to have at least 1,000 contacts on your list. That’s large enough to run a 50/50 split test with statistically significant results.
Now, if your list is much larger than that (say, 10,000), you can split up the test ratio differently. Maybe you’re testing a “jokey” subject line and you’re not sure it will have good results. If you have a large list, you can do a 90/10 split. The 10% who receive the “risky” test is still an appropriate sample size. And if the response is poor, at least it didn’t go out to half your contacts. But if the results are good, then you can feel more confident sending it to a bigger portion of your list.
Even if your list is smaller than 1,000, you can still A/B test. You just have to run the same test multiple times.
WHAT SHOULD YOU A/B TEST IN YOUR EMAILS?
Subject line
A subject line is your email’s first impression. It’s the make-or-break moment in your inbox that make you click… or hit the trash button. Sure, you can get tips on what increases response in a subject line from a variety of sources. You can even read important Do’s & Don’ts of Email Subject Lines on the J&C blog. But the best way to know what works for your audience is to test them yourself. Why? Because every audience is different. An email list that gets technical B2B communications will likely respond to different messaging than the list of a consumer product.
When it comes to subject lines, you can test a variety of things. Does adding an emoji increase open rates? How about a number in the subject line? Or phrasing it as a question? The more testing you do, the more data you have… and that helps you understand your audience.
Personalization
Personalization can improve the performance of an email. But what kind of personalization works best for your contacts? You could run an A/B test to compare the use of a recipient’s first name (e.g. “Hello Joe,” as the salutation) against a title and last name (e.g. “Hello Mr. Smith”).
Offer
Sometimes it’s not the words you use… it’s the offer you promote. If you have the resources, test specific offers against each other. For example, you could offer a 10% discount to one group and free shipping to another. Does one of these offers perform better than the other? This kind of test can help guide you to the promotion that gets your business the best response.
Images
Try A/B testing the hero image in your email. You could run a head-to-head test of an illustrated image against a photograph — or even a lightweight GIF. Or look at what the picture contains. Maybe your audience will respond more to an image of a woman using your product over a man using it. No matter what your hypothesis is, you’ll have to run a test to find out.
WHAT METRICS SHOULD YOU USE?
After you run your test, you should review a few important metrics. Open rate is a big one, especially for subject lines. You’ll see if one subject line motivated more people to click on that email and read the rest. If you’re testing something in the body of the email, like the offer, you’ll want to look at the click-through rate to see how many people took action. Last but not least, understand your conversation rate. Did you get people to sign up for a mailing list or purchase a product? If you get a great conversion rate on a test, then send to even more contacts. After all, that’s what testing is for. Learn from it and use it to create more effective marketing emails that get the results you want.
WHAT SHOULD YOU DO IF YOU NEED HELP?
As experts in the science and art of customer engagement, J&C understands the ins and outs of A/B testing. If you need help building a successful email strategy, contact J&C today.