Split Testing To Gain Critical Insights: How to Approach Your Next Test

When attempting to figure out customer intent, it can feel like throwing a lot of marketing at the wall to see what sticks. But the act of studying what sticks might be more important than many realize. According to McKinsey, companies who pay attention to customer analytics are 23 times more likely to outperform competitors in acquiring new customers. 

So, how do you arrive at analytics that are powerful? One of the most scientific approaches to generating new data is split testing. 

The split test is a scientific approach to marketing, similar to the scientific method you learned about in high school: start with a hypothesis, test it, and weigh the results.

In this article, we will study the art (and science) of split testing and discover the best ways to rigorously test which of your marketing campaigns leave the greatest impact on your customers.

Let’s start with a definition: split testing runs two simultaneous marketing methods against each other to see which one works best. In a split test, the two methods—such as two versions of a landing page for the same project—might look significantly different. It’s one of the key ways to generate customer insights so your team can make data-driven decisions.

To many marketing teams, the terms split testing and A/B testing are synonymous. But there’s an important difference here: split testing is but one approach marketers can take during experimentation and testing. 

The right testing method you choose depends on the variables you need to test, as well as the time and resources you have at your disposal. 

In other words, a split test might compare two entirely different web pages. Other forms of A/B testing might compare two identical web pages with one variable (such as a headline) isolated for testing. Multivariate testing, another common form of testing, tests multiple variations of multiple elements in a programmatic and statistical manner.

Let’s dive deeper into the definitions of these three common types of testing to understand when you’d use each one:

If you have the resources and the traffic to handle testing multiple options at once, a multivariate test is a handy way to cut to the chase. If you’re optimizing a smaller offering, you might want to start with A/B tests.

On the other hand, you might run a split test to see which style of homepage outperforms the other, for example, before you nail down the specific messaging around your service or product. This is when you’d run a split test.

Split testing helps you understand what your customers are thinking, which every marketer knows is critical to success.

There are plenty of ways to achieve this, of course. For example, you could:

The brilliance of the split test, however, is that it requires none of the above. It relies on observation on what customers do rather than what they say.

You can only achieve so much with focus groups before the data becomes unreliable and non-predictive. This is because of a phenomenon based on built-in biases and poor memory recall that enables people to report different experiences to reality (usually without knowing they’re doing it). 

Audiences might “say” they have certain preferences, but what decisions do they really make when they’re sitting at their computer and considering a purchase?

That’s what split testing asks while mapping actual behavior rather than reported behavior.

Before we dive into how to do it, let’s get more specific about these digital marketing benefits. Here’s what split testing can do:

In the early stages, a split test might feel a bit like you’re trying a little bit of everything (video marketing campaigns or ebooks, a simplified website UX, new branding, etc.). Even early on, you can run your test to optimize the quality of the data you’ll glean. 

What are your website visitors trying to tell you? Here’s how you find out.

Your first step is diagnosis. Now’s the time to consider a basic question: what do you want to test?

To begin, look at your existing data whenever possible: surveys, heat maps, session replays, website analytics. Chances are, you’re already sitting on a mountain of data. Use these assets to your advantage and find where customers are already pointing you. This will help you make educated guesses about your marketing assets.

Next, define the asset you want to test. Are you testing:

It may sound like a lot at first, but think of the testing process as a funnel: you’re starting with the bigger stuff and working your way down. Split tests help you test the broad, big-picture messaging at the top. Later, you can use singular A/B tests to nail the details.

If you’re feeling stuck, there’s good news: you’re not reinventing the wheel. People have run split tests before, which means there are plenty of online platforms you can turn to. Let’s review some of the tools you can employ once you’ve decided on the asset you’re going to test:

Chances are, one or more of these tools will help you diagnose the specific problems you’re having. You can integrate these tools into a new split test and then sit back and wait for the results. But don’t start that test until you’ve learned some best practices first.

Remember: you’re essentially running a science experiment here. Any good science experiment requires controls and parameters to ensure the quality of your results. Here’s what you’ll have to keep in mind before the split test:

At this point, you probably have a handle on the tools you want to use, the questions you want to answer, and what a split test can tell you. There’s just one remaining variable to separate from the equation: you. 

Split testing takes time to build up a large sample size that will give you statistically significant data. That means, it requires some patience. Set your test and then step away and avoid stemming any decisions from the data until it’s finished.

Let’s say you’ve run the test after completing all the steps above. You’ve been careful about selecting the assets to test and about the structure of your test. You’ve allowed a large sample size to develop.

Then, when it comes time to review the results, customers seem split. You have approximately a 50/50 balance between the two versions of your asset.

Was it all a waste of time? Not necessarily. Consider the example of Steven Macdonald, who once ran an A/B test and noticed zero influence on his conversion rates. 

In fact, he learned, a test like this does provide you data: it tells you that the messaging you just tested doesn’t have a major impact on customers.

If your tests don’t show you any statistically significant differences, you might conclude one of two things:

If it’s the latter, prepare to go granular. You’ve established the baseline, and now it’s time to dive into the details. That’s what Macdonald did, adhering to the principle of “constant improvement.” Macdonald says:

“You cannot expect to replicate these results, but what you can replicate is the continuous optimization of testing winning variations against newer variations based on the data you collect. Just because you run one A/B test, doesn’t mean it’s over.”

In other words, take on a “continuous optimization” mindset. Just because you ran one test and found little statistical variance doesn’t mean you’re done testing. Conversion rate optimization (CRO) isn’t always done with just one test. Now it’s time to zoom in and run a new test.

To run an effective A/B test, we need to start with a similar question: what are you testing? What makes something “testable” in the first place? For an A/B test, you’ll need to examine elements that are:

Some of the most common elements to test include headlines, headline positioning, colors, page copy length, sign-up workflows, checkout processes, pricing, and email subject lines. You aren’t limited to these elements, but that should give you an idea of what kind of variables meet the criteria above.

Finally, employ the same approach as you did for split testing. By now, you should have some familiarity with the tools possible, such as Optimizely. Many of these tools have features for A/B testing. Get familiar with the platforms and consider testing new colors, new copy, new pricing, and more. 

Continue to engage the best practices here: large sample sizes, and predefined schedules to ensure you stick to a test to give each variation a chance to prove itself.

Marketing decisions should be data-driven decisions. Marketers need to understand the buttons or features that keep conversion rates down or drive them up. With split testing, you’ll abandon the “shot-in-the-dark” mentality and take a scientific approach to figuring out what your customers want.

You might not be wearing a lab coat when you run a split test, but your findings will still be impactful. Remember to stick to your plan, gather a large sample size, and let the data show you where your marketing needs to go.