The online testing market is ripe for disruption. Site testing and optimization market has remained virtually unchanged since it was introduced more than a decade ago.

Compare the Forrester Wave: Online Testing Platforms, Q3 2015 report (paywall) to the previous one, released two years earlier — most of the players are the same and many of the overall approaches to testing are virtually the same.

What it comes down to is this: the traditional methods many companies use to test various parts of their site (e.g., AB and multivariate testing) have failed to evolve while the rest of the world moved forward.

It Takes More Than Testing

Marketers’ holy grail isn’t a certain level of testing capability, but a continuously optimized customer experience delivery that drives short term revenues and long term loyalty. Launching a test that compares a blue banner to a yellow one to see which performs better won’t prove this. It requires a comprehensive understanding of customers through the creation of a dynamic record of all of a customer's previous touchpoints and preferences. 

It takes more than testing to measure and analyze every single touchstone for each site visitor to create this record. These “touches” that build the record are how marketers gain deep insight and a comprehensive understanding of their customers. And it is this that enables marketers to build a lasting relationship through targeted personalization — and which absolutely cannot be accomplished through outdated approaches to site testing.

On its own, even the very best testing capability is insufficient to meet the needs of consumers. A lot of businesses invest time and a generous amount of capital on numerous testing projects. They sit in a vacuum with no data and come up with ideas of what they should test. Inevitably they end up testing small UI changes that have little effect. Our own research has shown that testing UI changes in 99 percent of cases has less than 1 percent impact on conversion rate. The average UI change increases conversion rate by less than 0.2 percent. These statistics do not indicate success any way you slice them.

Why? Because generally, the older approach to site testing includes comparing one or more variations of site content using non data-driven AB testing. Perhaps even more counterproductive, marketers are also using slightly more complex multivariate testing and throwing multiple, but random variations, into the test. Predictably, the winning variation ends up being a statistical fluke, just dumb luck really. Either way, only using AB or multivariate testing to understand customer behavior is missing the point entirely because human behavior and interaction is far too complex for online testing to grasp — and measure.

Furthermore, when asking brands what they want from online testing it’s always going to feel like a closed question where clients ask for “faster horses” rather than “cars.” By definition, it is a backward-facing process because as any online testing professional should be able to tell you, clients often don’t know what they want until you’ve invented it with them.

Testing remains just one element of the process by which the industry drives real-time customer insights. Once you have the insights and analytics to figure out who your customers are, what stage of the journey they are at and what they need to get to the next stage of the journey, etc., from there it should be straightforward to dream up personalized solutions. Your hypotheses should be based on what you already know and then verified with a test. Hence, testing should be used merely as a validation step to personalization and deeper customer experience that drives short term revenues and long term loyalty.

The Challenge Ahead

We, as an industry, have a ways to go before changing the perception that online testing is the most effective means to improve the customer experience. It might take a while for the market to catch up, however, this industry is becoming increasingly impacted by external factors that are making the context broader and the market more complex.

We need an industry-wide audit of the effectiveness of online AB testing and multivariate testing products and methodologies. Together we can measure and properly benchmark the field of online testing in context of the broader industry insights in the way that the advertising industry has done. The audit will address:

  • The current state of online testing and where it falls short of measuring the true customer experience
  • The critical components of the customer experience that testing misses entirely
  • How a holistic approach to measuring and managing the customer experience is complex and multilayered, mirroring consumers themselves

It is not surprising that testing is not — and has never been — a multibillion dollar industry. With the additional context that is being made possible and available through digital interactions with consumers, online testing really should be wrapped into the field of data-driven personalization. That’s the next $20 billion market and that is where the industry should be focusing its efforts in order to reap the bigger rewards.

Creative Commons Creative Commons Attribution 2.0 Generic License  Title image by  wwarby