blurry people and clear statue
PHOTO: Loïc Fürhoff

To truly understand customers it is essential to combine insights from the following research methods:

  • Quantitative
  • Observational
  • Qualitative

Quantitative data will tell you what is happening but only if it is configured properly, which often it is not. Bots and other sorts of unpredictable activity can seriously undermine the credibility of the statistics. For example, I once saw site whose most popular page had a throw-away reference to a Beatles song.

Quantitative data will tell you the current state, but it will rarely indicate what is missing, or unearth new opportunities and tasks that customers wish to complete.

How many visitors? How long did they stay? How many pages did they look at? What do these stats actually tell you? The worse your service is, the more visitors you may have to your website looking for support. The more confusing your menus and links are, the more pages they may be looking at. The more difficult your content is to understand, the longer they may be spending reading it.

Another problem with quantitative data is that humans are programmed to be impressed by size. Whenever you find a website in crisis you can be almost certain that the Cult of Volume was a major underlying cause.

The Cult of Volume ensures that so many digital teams are forced to chase hits, to create more apps, more sites, more features, more content. To get more visitors, staying longer, looking at more.

Search data is an important form of quantitative data but it has many weaknesses. The worse organized a website is, the more likely people are to search for top tasks. The easier to use a website is, the more likely customers are to use the navigation (because the top tasks will be prominently displayed), and only search for tiny tasks.

An extreme example of this behavior happened on the BBC intranet years ago. On the intranet homepage was introduced a section called Top 10 Searches. Employees began to click on these links and thus stopped searching for them. Six months later, the team noticed that there was a new top 10 searches, and so they replaced the old top 10 with the new top 10. What happened? The old top 10 became the new top 10 searches again.

Another factor is search maturity. When someone starts out in an environment, they will tend to do lots of searching. As they get to know their environment better, they will develop a list of favorite places which they will start going back to. Over a period of time, their search behavior is thus likely to change substantially, with less and less search for their top tasks. For example, a 2018 NPR/Marist survey found that 44 percent of US adults go straight to Amazon when they want to buy a retail product.

The words people search with does not always describe the task they are trying to complete. On the Microsoft Excel website, lots of people were searching for “remove conditional formatting.” Microsoft created a page to explain how to do this but this page never worked. Because what people really wanted to do was figure out how to use conditional formatting properly.

How did Microsoft know that's what people really wanted to do? Just like all the best digital teams, they began combining qualitative and observational data with quantitative data.