Case Studies

Smart Panda Labs had been working with Viceroy Hotels and Resorts—known worldwide for its luxurious hideaways, beachfront properties and five-star service—providing ongoing support for their Adobe Analytics implementation. When our team suggested adding A/B testing to the mix, some at the hotel group raised an eyebrow. They wanted the certainty of ROI. So our smart pandas established a two-month trial period for a series of tests using Optimizely – an industry leading tool – that would provide key insights into the client’s digital strategy, while driving new revenue along the way

As early as kindergarten, we learn how and why to run experiments. Do toothpicks float? What’s the fastest paper airplane? How can you drop an egg safely to the ground? We are taught the scientific method of asking and answering questions by making observations, forming hypotheses, doing experiments and analyzing the results.

Sometimes, however, adults forget the all-important lessons they learned as children. While our smart pandas know that the value of A/B testing (pitting variations of a digital campaign against one another) can be extraordinary, sometimes our clients ask us to prove it.

And so, we do. Here are the tests, and their results.

Right out of the gate, a quick and simple button copy test had tremendous results. We focused on a button on the hotel group’s website that acted as a “submit” button and linked to their booking engine. Our overarching goal was to optimize conversions by tweaking the button copy. To that end, we pitted five variations against each another:

A: Reserve (this was the existing copy and acted as the “control”)

B: Start Your Reservation (this was our “assumption”—we hypothesized this would come out the winner)

C: Make Your Reservation

D: Reserve Your Room

E: Book Your Room

The winner, by a length? E! Book Your Room.

But we weren’t done. We then iterated on that test by pitting Book your room against a slightly more descriptive alternative: Check Availability. The latter won by a long lead and, when implemented, increased room reservations by $30,000 per month.

A slightly larger test focused on guests and prospective guests of Viceroy’s Miami property. Their existing paid search campaign for the property was sending users directly from ads to the hotel website home page.

Our smart pandas looked at the website analytics to see where users went while on the site. The most valuable pages appeared to be the gallery, overview and promotions pages. We formed a hypothesis that a landing page that combined the best aspects of those three pages would drive more conversions via the booking tool than the original home page.

For the test, we sent 50% of paid search clicks to the control (the website) and 50% to the variable (the custom landing page). Our hypothesis proved correct. The landing page achieved a 32% increase in conversions and $600,000 in incremental revenue for the year.

So how’s that for ROI?

We further recommended that landing page templates be built for each hotel. As a second phase, we suggested targeting pay-per-click ad copy to a page that can dynamically align itself with the ad copy messaging.

For Viceroy Anguilla, we tested the addition of a promotional tile on the photo gallery page that linked to the highly-visited special offers page. We wanted visitors to have more options to reach the special offers page—one of the key pages of the site—than the existing navigational element. We hypothesized that this increased traffic would generate increased conversions.

In the first phase of this test, we pitted the control (no promotional element) against a generic tile promoting the special offers page. The promotional tile worked, increasing conversion as expected. If applied to the photo gallery pages across all hotels, we determined this would deliver an incremental $100,000 in revenue across a 12-month period.

The ancillary benefit of this test was proving that value can be attached to the promotional space on the website. Leveraging A/B testing, marketing teams can learn which promotions and specific messages in this space offer the greatest financial value.

On the Viceroy Miami site, our team identified several pages (Overview, Amenities, Accommodations and Rooms) to which we added either a link or a button that read “Make Your Reservation.” Our assumption was that we could increase conversions by more consistently inviting visitors to take an action. We tested the control (no interior calls to action) against “Make Your Reservation” links and buttons.

With a 95.2% level of confidence, the button version of the test increased conversion the most. This increase, extrapolated across all hotels and annualized, should result in approximately $300,000 in incremental revenue.

These relatively simple tests proved their worth, as most tests do. In fact, this testing trial resulted in optimizations that would drive more than $1M in incremental revenue for the hotel group. These results proved the value of testing to the client and more than delivered the ROI they were seeking.

A/B testing is a scientific method that places data at the heart of your digital decision-making. It is an invaluable strategy that trumps assumptions with real information, in real time. At Smart Panda Labs, we’re in constant pursuit of powerful data. Tapping into our A/B testing expertise can help you get the greatest return possible on your digital investments. To learn more about testing methods and value, download our free whitepaper, data-driven decision making with a/b testing.