A/B testing and multivariate testing both have their purposes, so how can you choose which to use?
In marketing circles, the classic A/B test has a long and honorable history. It’s been around since the 1930s, when mail-order advertisers would routinely run tests in magazines to see which version of an ad pulled the most coupons. Back then, of course, testing the efficiency of one element at a time was sufficient. Now, online marketers need to measure multiple variables -- all at once and in relation to one another -- to coax more sales and conversions from their websites. This can’t be achieved by A/B testing. Hence the emergence of multivariate (MVT) testing.
So which one is right for you? Well, it depends on what you’re testing and what you need to know. Consider the following as you make your decision.
If you simply need to know which headline on your landing page will generate more click-throughs, downloads or sign-ups, A/B testing might be the right choice for you. But if you need to know that, as well as which price presentation delivers higher sales, and with what headline the winning price presentation performs better, you’re going to have a lot more success (and save a lot of time) using multivariate testing.
A/B testing is simple and easy to understand and the tests are relatively easy to conduct in-house (if you prefer). And it’s still better than guessing or going on instinct.
Marketers who live and die by their online results, however, need a more complex solution and therefore generally rely on multivariate testing. Multivariate testing applies to a multitude of page combinations, opening up many more possibilities for understanding how your site works for visitors, and how to continually refine it.
Here’s where multivariate testing pays its way.
MVT Reveals How Your Site Elements Interact
Unlike A/B testing, multivariate testing is particularly good for understanding the interplay between various elements of your customer experience. For example, let’s say you do an A/B test on two or three headlines for a landing page. You then take the “winner” and continue testing with two or three calls-to-action (CTA), perhaps, or different product images.
The trouble is -- as marketers discover all the time -- it’s entirely possible that the optimum solution is in fact your original headline but with a particular CTA and graphic combination. A/B testing can miss that entirely. Or take forever to figure out.
The multivariate approach lets you explore various combinations of your page elements at the same time -- headline, graphic, CTA, product ratings, navigation -- to ultimately arrive at the most productive solution. As often happens, an optimum combination of changes typically lifts conversions more than a single change of headline or visual.
MVT is Better for Exploring
Let’s say you’re faced with a homepage or a checkout sequence that doesn't seem to be performing well. What should you start working on? Where are the soft spots? A well-designed multivariate test allows you to experiment with dozens of elements to see which of them move the needle the most -- and in what combination. It’s not uncommon for marketers to run tests involving as many as 96 variants, trying to understand the dynamics of a page or sequence.
While that may sound dizzying, a multivariate test will rather quickly point to elements -- or combinations -- that have the greatest effect. (Or just as valuable, you’ll see which elements don’t affect much at all.) The results and patterns will lead you directly to a better-performing page, as well as inform your design and development decisions going forward. You understand more than you ever could with A/B tests.
In a recent test, a rental company discovered it could increase sales of “extras” by 25% by adjusting the size, positioning and volume of product information -- revealed by multivariate testing.
MVT Fosters a Test-and-Learn Culture
What would improve your multi-page checkout sequence? Do returning visitors respond differently than first-time visitors? Should your landing pages differ for customers coming from different sources? What should you change? Should you ask for more information on membership sign-ups? Or less?
The most successful digital marketers ask questions like these all the time -- and continually test, refine and optimize what they’re doing. It’s not about wholesale makeovers but continuous iteration, driven by constant experimentation. Using multivariate testing not only provides a platform for rapid exploration and validation but also creates an environment for creative teams, marketing and development to keep asking, “How can we make this better? What can we try, right now?”
That’s how the digital elite think.
Title Image courtesy of Lightspring (Shutterstock)
Editor's Note: To read more of Mark's tips for fine-tuning your website, see his Let Your Customers Optimize Your Website for You