Marketing Strategy

Testing email marketing: how to create your own guide to effective emailing.

a/b split testing a/b testing a/b testing email marketing a/b testing ideas email A/B testing

The higher the stakes are for email marketing in your business, the more controversies you’ll get about what your letters should look like. To try and pitch effectively, we focus on the portrait of the client, product knowledge, marketing trends, tips from professionals, and just general studies.
The safest way is to constantly test every idea you have, and react according to the response of your database. In this way you build your own guide with the most effective solutions for your business. Disputes are not necessary when there are numbers on hand.

Split testing and A/B testing

This is the main method to test your letters. The gist is, make multiple versions of a single letter and divide them by the testing segments. Specify the optimal conditions and the time frame (e.g., 3 hours). When the time passes, you’ll be able to determine the winning letter which then goes to the core part of the database.

How to:

    1. Form a precise hypothesis that you will either prove or disprove. Choose hypotheses that can later be converted into rules, e.g. “in a regular mailing throughout our database, personalization in the subject line increases its efficiency”.
    2. Prepare a letter and its alternative version based on your hypothesis (A/B test). In a split-test you can have two or more alternative versions. The letters should only differ by the one element that you are testing with your hypothesis.
    3. Set up the test. Usually mailing services offer ready-made templates for split tests. In SendPulse they look like this:


  1. Form a segment for your test. Usually test messages are sent to 20% of the database, and the winning letter automatically goes to the remaining 80%. At the same time, each of the test versions should be received by at least 500 people.
  2. Choose your success condition. If you’re testing the subject line, the letter with a higher open rate wins. If you’re testing main body elements, look out for the number of conversions. I recommend taking into account all parameters and selecting the winner manually. For example, if the subject line worked well but didn’t go with the content of the letter, you will be getting less conversions, despite a good open rate.
  3. Execute repeated hypothesis tests and make a conclusion. It’s best to test each hypothesis 3-4 times with an interval of 1-2 weeks. Check the statistics both immediately after the test (3 hours after sending) and the next day, or, even better, in 3-7 days. The results can be significantly different. You should take everything into account – open rates, conversion rates, target actions and the number of unsubscribed users. Take a note of the best solutions across all parameters and continue to use them in the future.

What’s worth testing?

In general, you should test everything.

Start with the most strategic elements: which content type and communication style works best? Do you need a branded design? Which templates are most efficient? What kind of offers, promotions, gifts and calls to action grab your audience of subscribers? This data will form the basis of your strategy.

Then start testing the marketing subtleties that will complement your strategy:

Subject Line

Test the length and the number of words in the subject line, CAPS LOCK, amplifier words, special characters, personalization by name and geolocation, question/call to action, digest / unique topics, digits, quotation marks, specific offers or generic wording.


Evaluate different sender name options – try company’s, manager’s, supervisor’s, project’s, service’s or fictional character’s names.


Here you can include your greeting message, the letter subject, the key proposal, a call to action, or just follow the natural flow of your topic.

Letter design

Test the design colours, your selection of pictures and banners, the colour, size and layout of the buttons, the design of in-text links, frames, branded elements, font style, colour and size.


It is interesting to analyse personalized greetings, the letter structure and length, formal or casual approach, amplifier words, various wording options, use of reviews, feedback request, client’s purchase history, signatures and footers.

Testing frequency

When you start seeing the effects of your tests – client’s interest or sale growth, you’ll be tempted to increase your mailing frequency. Most companies start with a weekly newsletter and then increase the number of emails to 3-4 a week; some even begin to send letters every day of the week, including weekends.

You can determine a comfortable mailing frequency manually. This kind of test takes at least 2 months. In general, the longer the test, the more accurate the results. If you have the resources, you can even take six months to carry out the experiment.


This is a manual test. In the database, choose 2 or more random segments. These segments should have similar average mailing performance indicators. To ensure this is the case, distribute mailings to all segments evenly, according to the old scheme for 2 weeks. If everything is fine, move on to testing.

Leave one segment aside as a control element. Email the rest of the segments same series of letters but with a different frequency. Check the performance indicators of each message for each segment and take note. As a bare minimum, check the open rates, conversions, target actions and the number of unsubscribed users.

After around 2 months check the results by segments. First check how many people unsubscribed: 64% of people unsubscribe from mailing lists because of a high frequency of emails. If you see an alarming trend (e.g. if the number of users unsubscribed increased by more than 3 times and continues to grow), it’s better not to distribute emails so often, even if you see an increase in sales. Think about how this will reduce your database in just a year and how much you will lose.

Then take a look at the dynamics, what’s happening with your open rates, conversion rates, and targeted actions. The number of openings may fall by about 10-20% – don’t worry, this is natural and you’ve already made up for it with additional mailings.

Check the total number of unique conversions in each segment, average revenue per subscriber and total income from each segment. If you see a significant result, transfer your database to the winning scheme and continue to closely monitor the dynamics.

Testing the distribution time and day

Any marketer will tell you that best days to distribute purchasing mailings are Tuesday, Wednesday or Thursday. That entertaining and informative content is best to be sent on Fridays and weekends. That you should send your mailings at 9-12 am (according to some theories, even at 5-6 in the morning).

For example, our Pardot colleagues’ recommendations looked like this just three years ago:


Have these trends changed and how do they relate to your audience? Form a hypothesis based on your client and test it, even if it is contrary to the main principles of marketing.

Use the same scheme to create split-tests. After the tests are over, stick to the chosen day, time and distribution frequency. Subscribers will be more comfortable that way, and it will be easier for you to plan your work schedule.

So what’s the bottom line?

  1. Testing is mandatory.
  2. Be extra careful when testing distributions with massive databases.
  3. Test one element at a time. Prove each hypothesis at least 3-4 times.
  4. Test only what you can use in the future, and keep improving your distributions by using verified methods.
  5. Start testing the strategic elements of your content, and then move on to testing subtleties.
  6. If you choose to increase the frequency of your mailings, test carefully and don’t forget to check the statistics. Take into account the total number of clicks and sales from each segment. Be extra attentive to unsubscribing dynamics.
  7. Days of the week and distribution time should be tested based on the unique features of your audience.

Wishing you successful testing and effective emailing!

Testing email marketing: how to create your own guide to effective emailing.

SendPulse is an email service with additional channels: web push notifications, SMS and Viber

Up to 15,000 emails for free every month. Web push notifications are completely free.

Sign up for a free accountHow does it work

Send me new articles by email