According to the a new Forrester Wave report, online testing technology is undergoing a rapid evolution where it is becoming a core competency of digital marketers, while leading online testing platform providers are starting to follow the lead of digital marketing vendors by offering functionality that support multiple channels and create consumer profiles.


Online testing is turning into a core digital marketing competency because of the valuable support it can offer optimization efforts. Advanced online testing solutions evaluate customer experiences against a control group using statistical data to help marketers determine how to provide an optimal customer experience.

In the report, Forrester Wave Online Testing Platforms Q1 13, Forrester identifies several “next generation” capabilities provided by mature online testing technology providers. These include distributed access to support a wide range of users who may not be technical experts (such as marketers) as well as to support the rigors of online testing activities, sophisticated multichannel testing capabilities, the ability to collect, store and distribute visitor data at a granular level, advanced testing scenarios that encompass multiple pages, domains and sophisticated success metrics, automated core testing processes and analytics and integrated suites of optimization tools.

Forrester identifies seven online testing vendors as standouts in the field. In order of rank, they are Maxymiser, Adobe, Autonomy, SiteSpect, Webtrends, Monetate and Optimizely. Following are summaries of Forrester’s review of each vendor, divided into the “Leader” and “Strong Performer” categories.

The Leaders - Maxymiser, Adobe, Autonomy, SiteSpect

Maxymiser receives top honors among all online testing vendors from Forrester due to its “consistently strong capabilities across all categories of online testing functionality.” In particular, Forrester singles out Maxymiser’s reporting and analysis functionality, as well as mobile and social testing. Furthermore, Maxymiser receives credit for focusing on the US market to the point of relocating executive management to New York despite being a UK-based company.

Forrester says users who need a balance of professional services and advanced technology will most benefit from Maxymiser, but cautions the vendor must “execute on its vision for comprehensive optimization and visitor data management” to maintain its category lead.

The Test & Target component of the Adobe Marketing Cloud suite is the runner-up among online testing solutions reviewed by Forrester, who calls it a “solid product across all facets of online testing capability.” Test & Target also benefits from Adobe’s strong ecosystem of third-party partners and existing integration with commonly used Adobe analytics and content management technology. Forrester says current Adobe users have most reason to use the application and advises Adobe to improve ease of implementation and speed up its product development timetable.

Learning Opportunities

The Optimost component of the Autonomy Promote suite comes in third. Running on the Autonomy Intelligent Data Operating Layer (IDOL) analytics platform, Optimost gets kudos for “strong testing capabilities featuring a variety of optimized techniques.” Forrester expects users with highly advanced testing requirements to get the most value from Autonomy Optimost and suggests that Autonomy update a “badly dated” user interface and “address ongoing market uncertainty” as a result of its 2011 acquisition by HP.

Although SiteSpect gets attention for its tagless deployment model, which Forrester deems a “polarizing” market differentiator, the report praises its “very strong testing capabilities” and especially recommends the solution to technical users. SiteSpect needs to improve usability and capitalize on the advantage tagless deployment offers in testing emerging channels such as mobile.

The Strong Performers - Webtrends, Monetate, Optimizely

The Optimize application from the Webtrends social/mobile/Web analytics platform is the highest-ranked strong performer due to its “rigorous, statistically trustworthy” approach that is sometimes mis-characterized as inflexible. Webtrends is most suited to users of managed services for advanced online testing and the vendor is advised to rely less on professional services providers and more on integration with its other products.

Monetate receives recognition for its “meteoric” rise from a startup two years ago to a well-funded major online testing competitor today. It offers strong functionality and ease of use, resulting in high customer satisfaction scores. Monetate is particularly aimed at e-commerce and retail users. However, to remain competitive, Forrester says Monetate must “expand its industry footprint, continue developing advanced testing features, and expand its support and services resources to keep up with customer growth.”

Optimizely is the newest company to be included in Forrester’s rankings and has risen from a “cheap, no-frills A/B testing tool” used by smaller businesses to a more widely deployed tool for enterprises of all sizes. It is designed for marketing-oriented, rather than technically-oriented, users. Forrester lauds Optimizely’s “ethos of simplicity” and “modern, highly intuitive” user interface. Despite these advantages, Forrester says Optimizely needs to expand its “enterprise credibility” with more professional services and support and advanced testing features