3 Ways to Get More from A/B Testing

    1024 512 Shamir Duverseau

    3 ways to maximize the value of your a/b testing platform

    “Why isn’t everyone doing this?” That was the overwhelming thought that went through my head the first time I was introduced to A/B testing almost a decade ago. Now, 10 years later, I am still in awe of the number of organizations that have yet to implement an optimization program. From the local florist, to the Fortune 100 company, everyone’s goals are fundamentally the same, to increase revenue, to reduce costs and to improve customer loyalty. An optimization program helps you accomplish all those things on an ongoing basis, so why isn’t everyone doing it?

    For a lot of people, the answer could be that they oversimplify A/B testing. Perhaps they think it’s all about trying out new buttons and headlines and figure that gets old fast and just isn’t enough to justify the investment. Starting and running an optimization program is an expense —from the software, like Optimizely, to the internal or external resources required—and expenses require justification and ROI. And when each test is, frankly, a bit of a gamble, ROI isn’t guaranteed. While every test will provide insight, it won’t always improve the business, and that can be hard to stomach.

    As you might suspect, I would say they’re looking at this all wrong. Instead of seeing A/B testing as a comparison tool for buttons and headlines, businesses should look at testing as a problem solving tool. Here are three examples of how A/B testing and optimization can make an impact on your business, and your bottom line.

    new products and promotions

    Whether you are trying to improve upon an existing product or bringing a new product to market, how can you know whether your offering will actually improve your business?

    A/B testing can be utilized to introduce the product or service on your website or mobile application. Conduct a 50/50 test across your entire audience—or segment of it, by isolating a fraction of your audience and only testing them. Now you have an opportunity to measure how people are responding to the new product and whether that response is affecting sales, profit, marketing costs or even your other products.

    The same principle applies for promotions. From free shipping to discounts and value-adds, the goal of promotions is to increase sales—without sacrificing profitability. In other words, you want to generate enough incremental sales to more than compensate for the additional costs generated by the promotion. Running the promotion as an A/B test will shed light on factors like conversion and profitability as well as audience response. You might learn that the promotion is more cost effective in some channels or geographies and less so in others. This information will help you run that promotion in a more targeted and effective way.

    testing-word-cloud

    customer feedback

    AB testing and can help increase the lifetime value of your customer relationships. Using tools like Foresee, you can understand the level of satisfaction people have with your website or mobile app, both overall and in relation to specific areas like navigation and site information. Moreover, you can leverage these voice-of-customer tools to see how those satisfaction measures affect the likelihood of a person returning to your site or referring your business to someone else.

    foresee-cx-survey-sampleA sample report from Foresee’s CX Survey showing satisfactions scores and their impacts on future behaviors.

    If you learn that a significant number of people are dissatisfied with your website’s navigation, for example, and their dissatisfaction is affecting their likelihood to return or refer, you know it’s time to test. You can toll out an A/B test of your navigation, pitting two or more versions against one another. Then, using the same voice-of-customer tools, you can judge the test against key performance indicators like conversion, sales and leads. You can also learn how the variations affect satisfaction.

    development and technology updates

    Your development or IT teams have to make continual changes to your business’s website or mobile application, often to update or add enhanced functionality. However, these updates often present challenges because people don’t like change, and you can’t be sure how these changes will affect the overall business. Unless you have the budget to conduct robust usability labs and focus groups, there is always the chance that a change in your technology aimed at improving performance could end up backfiring.

    Enter A/B testing. Before making a global change, test the new functionality. Before updating your hotel booking engine for everyone, for example, update it for a fraction of your site visitors and then throttle the amount of traffic. This will enable you to measure the effect of the new engine in real time, analyze how different audience segments are responding, and even isolate problems (e.g., why is everyone converting well except for visitors using Safari?). If and when the problems outweigh the benefits, you can turn off the test and revert back to the original engine.

    And what about that update that needs to be made to the mobile app, submitted to the Play and App Stores and then pushed out to everyone? Depending on the change, test it to ensure that it will have the desired effect. If it does, now you have data to justify the time and expense of updating the app. If it does not, you just saved time and money. Either way, you win.

    testing-correlation-gridWhile older data, notice the correlation between high ratings and frequent updates.

    Have some new technology you are considering investing in and adding to the website? See if you can test it first to gauge the response. Ask for a proof of concept and put it through an A/B test to see what measurable difference the new technology will make. With page level voice-of-customer tools, engagement and conversion rates, you can get a feel for the new technology’s ROI.

    A/B testing and the tools to implement them can help you solve problems in ways that can be measured and analyzed. The value of A/B testing and optimization extends well beyond a button test. Granted, the ideas we have discussed are also more complex to plan, build and execute than creating variations of a call to action. However, the intel they provide can make a real and profound impact on your business by taking the guesswork out of many of the decisions you will inevitably need to make anyway.

    So, if you don’t have a testing program in place, maybe it’s time to ask yourself, why not?

    key takeaways

    • A testing and optimization program can help you increase revenue, reduce costs and improve customer loyalty.
    • A split test can measure how people are responding to a new product or service and whether that response is effecting sales, profit, marketing costs or even your other products.
    • AB testing, when used in concert with voice-of-customer tools like Foresee, can help you measurably improve the level of satisfaction people have with your website or mobile app— and if their improved satisfaction is positively affecting their likelihood to return to or refer your business.
    • Running a promotion as an A/B test will shed light on factors like conversion, profitability and audience response as well as help you run that promotion in a more targeted and effective way.
    • A change in your technology aimed at improving performance could end up backfiring. Before making a global change, test the new functionality. If and when the problems outweigh the benefits, you can turn off the test and revert back to the original engine.
    • The intel A/B testing platforms provide can make a real and profound impact on your business by taking the guesswork out of many of the decisions you will inevitably need to make.

    newslettersignup-promo

    AUTHOR

    Shamir Duverseau

    Shamir is responsible for business management and leads the firm’s analytics + intelligence and conversion rate optimization practices. He has worked across a number of industries, from travel to entertainment to technology, working with brands like Southwest Airlines, The Walt Disney Company, and NBC Universal. During his last 15 years in marketing Shamir has held leadership roles, overseeing everything from product management to digital strategy, including user experience design, web development, testing and web analytics. Prior to smart panda labs, Shamir was the Senior Director of Digital Strategy and Services for Marriott International’s Vacation Club Division.

    All stories by: Shamir Duverseau

    Leave a Reply

    Your email address will not be published.