A/B testing your emails is an important aspect of great marketing.
Rather than assuming what your market wants or prefers, it offers you the chance to find out which version they respond better to, which will in turn help you achieve your goals for the marketing campaign.
This guide contains everything you need to know about A/B testing emails in Sitecore.
Let’s get into it.
On the Sitecore launchpad, click Email Experience Manager.
Select Regular Email Campaign
Choose the template you want to use, and then click Create.
Complete all fields on the General tab:
Click on the Recipient tab
Choose the list you want to send emails to with the Include List dropdown.
If you want to exclude any recipients from the send, which might already be in the Include List, you can add their email to a suppression list and select it in the Exclude List dropdown (optional).
Click the Message tab.
You’ll see that you’re editing message variant A.
Set the email subject, content, and the alternative text version of the email (the version that recipients will see if they can’t view the HTML version.
To change only one thing between the A & B version (recommended):
Click the copy symbol.
To set a completely different content as the B version: (not recommended).
Add an additional variant (the B version), by clicking the plus sign. NOTE: You can have more variants than just the A/B, but this is not recommended.
You’ll now see Variant B appear next to Variant A, and can update the email body, subject and alternative text versions.
To remove a variant, select the variant you want to remove, and then click the dropdown arrow, and select Remove this variant.
To preview how a version will look for recipients, select the version, and then click the eye icon. Choose a recipient from the list and click OK.
Click on the Delivery tab.
Select which versions you want to include in the send. If you’re doing an A/B test, you want to select both of them (or all if you’re doing a multivariate test - not recommended).
Select the ‘Size of the test’ by choosing a percentage from 1-100%. This determines what percentage of recipients will be included in the test. If you choose 10%, only 10% of contacts in the list will enter the A/B test.
The rest of the list will see the winning version (if you choose to select a winner automatically. If you choose to select a winner manually, the A/B test will only send to 10% of contacts in the list, and then you will have to choose the winner manually).
Choose whether you want to select a winner automatically, or manually.
NOTE: If you choose to select a winner automatically, you will not be able to view the final numbers that the winner won by. Sitecore will choose the winner for you, and then send the rest of the emails to the winning version and you will never know how much version A or B was successful by.
If you choose to select the winner automatically: You can choose the amount of time you want the winner to be selected after and whether you want the winner to be chosen based on the open rate (a test of the subject lines), or the value per visit.
If you choose to select the winner manually:You’ll have to return back to this area to choose the winner based on either the open rate, click rate, value or action.
Choose whether you want to send the email now, schedule the delivery for a specific date or time in the future, or if you want to send it recurrently.
If you want to be notified when message delivery is complete, click the checkbox.
If your email has personalized content, click the Personalization checkbox.
If your email is a service campaign, click the Service email campaign checkbox (so all recipients receive the email regardless of their defined consent settings).
If you have multiple languages configured in your Sitecore instance, you can select the Multi-Language checkbox, which will send the message in the preferred language of each recipient.
To start the A/B test, click Start A/B test, and then click OK.
As mentioned above, the only way to view the results (in numbers) of your campaign, is to choose to select the winner manually.
Click on Email Experience Manager from the Sitecore launchpad.
Click In Progress.
Here you’ll be able to see all the A/B tests you have running at the moment, that are waiting for you to choose the winner.
Select the email that you want to choose the winner of, and click the Open campaign icon.
Click on the delivery tab
You will see a table that shows the open rate, click rate, value and action for each variant of the email.
Click the Choose as winner button, for the variant that you wish to make the winner.
The rest of the contacts in the campaign will be sent the winning version of the emails.
If you chose to select the winner of the A/B test automatically, Sitecore will send the emails to the size of the test pool you decided on when you chose the % of recipients for the test, and Sitecore chose the winner after the amount of hours that you set, based on the criteria you set the winner to be chosen by.
Sending A/B test emails is easy in Sitecore.
The aspect of A/B test emails that I get asked the most questions about is how to view the results of the A/B test, and whether you should choose the winner manually or automatically.
I hope this blog gave you enough insight into whether you should select the manual or automatic winner option depending on what you’re trying to achieve with the campaign.
If you want to see the exact numbers for each variant of the email in the test, choose to select the winner manually.
If you aren’t interested in seeing the exact numbers for each, and would like Sitecore to choose the winner for you, select the automatic option, but keep in mind that you won’t be able to see the full results of the test, other than knowing which version won.
To find out how to set A/B page tests up, read my blog about Running A/B and Multivariate Tests in Sitecore.
If you have any questions about A/B tests, emails or any and all things marketing and Sitecore, please feel free to ask them in my Facebook group for Sitecore marketers and content authors, or ask me on Twitter @natmankowski and I'll reply to you personally.
Sign up to our bi-weekly newsletter for a bite-sized curation of valuable insight from the Sitecore community.