An introduction to A/B split testing

It got me thinking, how much do we hold back so we don’t fail? How good could we be if we push ourselves to the point of failure, learn our limits, and then get back up again and carry on with renewed knowledge and confidence? Failure can be a better teacher than success.

I am not advocating reckless actions and taking risks for the sake of it. But what would happen if we pushed ourselves further to find out if we have a good idea or initiative?

It’s ok to fail at the gym, it’s a safe place to fail in. Not too much is at stake. But what about advertising? Advertising can be costly and failure can be expensive. Is there a safe place to be a successful failure in advertising?

A history of failure to win

Claude Hopkins was one of the successful great grandfathers of advertising. He was a big advocate of experimental advertising. He would take his best performing advertising and beat it with incremental improvements. Claude called it Scientific Advertising. It was a process of success and failure. He risked failure in small amounts to create more successful and effective advertising. He said “It is not uncommon for a change in the headline to multiply returns from five to ten times over”. Sometimes changing a few words around in your headline can have a huge impact on the number of sales and conversions. You have to test a few different headlines until you find something that converts well.

What Claude was doing over 100 years ago is now called A/B Split Testing. This approach is a bit like an ad contest. You show half of your customer audience one ad and the other half a slightly different ad. You eliminate the worst performing ad and then introduce another ad to try and beat the hero. The process goes on until your best ad is difficult to beat. You can then take your winning ad to a wider audience with a larger budget, with confidence.

Your safe place to fail

Today, digital advertising is the perfect platform for developing powerful advertising messages. It allows us to take lots of small risks without losing too much money. We can test one headline, offer or message against another. For example, when we run Facebook and Google advertising campaigns we test safe headlines against more risky ones with the aim of getting more attention and better results. We test one benefit against another, one offer against another and so on. We keep trying new (sometimes risky) headlines until we beat the existing headlines. We measure success as the number of people who take our desired actions (conversions). A desired action could be filling out a form, clicking a button, making a phone call or completing a purchase online. You have to be ready to fail to succeed with this approach, but in the long run, you win.

A/B testing can also be used on website pages to discover the most effective page to achieve your goals. You can test a multitude of factors. A good place to start is with testing the headline, page colour, main image, offer or call to action. You don’t want to test all these things at once. You choose one thing to change up and then check the results and then add another thing to change and so on.

Of course this thinking can be used in traditional advertising too. In fact, this is where the strategy was invented.

I once did a direct mail campaign for a bank targeting large commercial customers. As a first step, we were trying to test an offer to see what prize was more appealing to our audience. We sent one group the offer of the chance to win a holiday if they booked a meeting with a Bank Manager to discuss their finance. We offered the other group the chance to go in the draw to win cash. The cash prize won. We then went to a wider group with the cash prize draw offer and had a successful response.

A/B Testing can also be used when a team can’t decide on which ad should be run. There can be two or three strong ad ideas on the table and the team can’t decide which one is best. You don’t need to lose sleep over this. Put the ideas to the test on a small group and let the customers decide which is best before you roll it out to the bigger audience.

A process to follow:

1. Define your goal. What specific result do you want to improve?
2. Create the Challenger. Design the idea that you think will beat the current winning champion ad or web page
3. Prioritise the variables – Step two often delivers a number of possible ideas. Narrow it down to one thing to test.
4. Run the test
5. Measure the results and review
6. Start the process again

How does it work practically

If you are running the A/B test on a digital platform like Facebook or Google they make it easy to run multiple ad ideas. If you want to run the test on your website then most of the major website platforms have extra tools that allow you to do this. There is often more to setting this up than in Facebook or Google so you may need the assistance of a Web Developer to do this. If you wanting to use traditional media to run split test then you will need to use other strategies like coupon codes and offers where you have to enter a code to allow you to track and measure the results.