As a poor college student, I would have done nearly anything for an extra buck. So when I was offered $20 to be observed while trying to accomplish various tasks on a 1990s website, I jumped at the chance.

Being young and eager to please, I worked hard to find errors and problems on the site. The observers had asked me to vocalize my thought process, so over the course of an hour, I dredged up enough nitpicking issues to make Oscar the Grouch proud.

Finding Real Versus Invented Problems

Looking back, I realize that although many of the problems I identified really were annoying, some of the issues I flagged in the lab would never have bothered me in a real-world setting. The flip side was true as well: There were probably some very real site issues that I never noticed, since I was following specific instructions about what to try.

In short, my usability test did not accurately reflect what my actual user experience would have been had the testers left me to explore the site on my own.

The Limits of Usability Testing

This recollection got me to thinking recently about the merits and shortcomings of usability testing in today’s modern digital arena. Traditionally, usability testing has been used to give us insight into customers’ visceral and emotional reactions to branding and design, as well as their more generalized subjective responses to the holistic onsite experience.

Yet the massive scale, agility and responsiveness to design subtleties that characterize today’s e-commerce sites make it nearly impossible — and way too costly — for marketers to keep up with their customers through traditional usability testing alone.

The Power of Customer Experience Management

Today, brands in search of a truly accurate reflection of their customers’ needs and intentions require a more comprehensive customer experience management (CXM) approach that lets brands track the digital body language of customers as they actually behave online in their “natural habitats.”

What’s more, only by viewing that “untamed” user experience in the aggregate can brands gain a truly accurate sense of what’s resonating — or isn’t. Think of it as the difference between watching lazy, well-fed lions at the zoo versus being on safari and observing them hunting in the wild.

Only the safari delivers the big picture necessary to uncover the frequency and severity of usability flaws — while also identifying potential opportunities to achieve higher levels of customer satisfaction.

Think Safari Not Zoo

Here’s a real-world example: A high-end consumer electronics manufacturer relied on a visually rich product page to telegraph its product benefits quickly to customers. To ensure that the advantages were as clear as possible, the company conducted traditional usability testing, as well as page evaluations using advanced user experience technology. 

During usability testing, when subjects were asked to perform a specific task on the product page they did so with a minimal amount of scrolling and zeroed in on the page’s video elements. These findings made sense to the company, given the good conversion rates it was already seeing from its videos. 

Good Enough Isn’t Good Enough

However, when a far more comprehensive population of visitors was monitored using advanced CXM techniques, a dramatically different conclusion emerged. A significant number of visitors were very interested in content that was located toward the bottom of the product page — so many in fact, that the bottom of the page was drawing more customers than the videos.

What’s more, aggregate data represented in heatmaps and confirmed by drill-downs into individual anonymous session replays revealed that many visitors skipped over the middle of the page to find content that appeared only at the end. 

The implications for page performance were clear: When the site removed its poorly performing mid-page content and moved interesting content up from the bottom to feature it more prominently beside the well-performing videos, overall page performance improved dramatically.

Subjectivity Injects Bias

Usability testing is inherently subjective, as the overly detailed feedback I gave during my college experiment richly illustrates. Much like my younger self, people in usability testing situations are often overly focused on pleasing the observer — who may be paying them — by generating lots of detailed feedback. 

In short, usability testers tend to be more stringent, observant and critical than they would actually be when shopping from their own computers or smartphones. That’s why it’s important to keep the difference between what people say affects them and what really affects them firmly in mind at all times .

Objectivity, Context and Scalability

As our case study shows, observing users anonymously can provide a far more complete and fair picture of actual usage scenarios. When users don’t know they’re being evaluated, there is no chance of “contaminating” results with user presuppositions, something that traditional usability testing simply cannot deliver. 

What’s more, on large-scale online retail sites, looking at user behavior on a single page ignores the broader context of the online experience — the overall customer journey. To truly understand why users behave in certain ways, w e need to take into account how they arrived at a page and where they are going next.

Finally, and perhaps most important, the sheer scale of digital commerce brings the cost-effectiveness of standalone usability testing into question. Conducting usability testing for 10 subjects three times a year carries a significant price tag. This same sum could be used to monitor literally a thousand times that number of users on a daily basis to drive the agile interaction design that is crucial to keeping conversions high. 

Let Your Objective Determine Your Approach

The bottom line is to understand your marketing objective before determining which approach makes sense. Let’s be clear: Usability testing plays a key role in measuring high-level, global and impression-based issues that may impact a brand, such as new logos or new product packaging.

Small-scale, face-to-face usability sessions can pick up on visceral reactions and observable responses that reflect the actual emotional reactions of the testing subjects, something that technology cannot yet do on an individual scale.

CXM Generates Higher ROI

However, on the global stage facing millions of digital consumers — and with billions of dollars in sales and shareholder value at stake — it’s important to know when a hybrid approach like digital customer experience management is the better choice.

Chances are the ROI of optimizing according to the aggregate data of millions of users will be far greater than the limited-scale improvements that can be achieved via a usability lab.