Evidence-based decision making is something that very much suits the Cisco culture. “Consider the alternatives to evidence-based decision making and it is hard not to be a believer,” Bill Skeet, senior manager of customer experience for Cisco Digital Support states. “If you aren't using evidence to make decisions, then decisions are subjective (rather than objective) and imposed by fiat. The evidence-based approach promises to be more scientific and therefore predictable and reliable.” 

In many models of web management, every two to three years the site gets a redesign. This could be called the "lipstick on a pig" approach to website management. The result is often that the site gets a nice new facelift but rarely that underlying problems are properly addressed.

The evidence-driven continuous improvement approach is key to the success of Cisco Support. One advantage is that it “allows for rapid course correction,” as Bill explains. “This allows teams to move faster and make occasional mistakes without long-term consequences. Unfortunately, many organizations have a history of executing monolithic projects in a 'launch and leave' fashion. Teams that want to embark on continuous improvement may have to overcome 1. lack of skills on the team, 2. financial processes and 3. a penchant for 'big bang' releases.”

The Top Tasks approach takes the focus away from projects and instead puts the focus on the customer’s ability to do the most important things they need to do. The team running the website are no longer measured based on the content, webpages or designs they produce but rather on whether they have made their customers’ lives easier.

Learning Opportunities

According to Bill that has had a “dramatic” impact on how people think about their jobs. “We now track the score of each task and set goals for each task. We have assigned tasks and goals to product managers to make sure we have a person responsible for managing the quality of the experience. We conduct a Task Performance Indicator (TPI) assessment every four months to track the impact of projects on the TPI score.”

A Task Performance Indicator is conducted as follows:

  1. 1. Based on the top tasks, task measurement questions are constructed. Here are some examples of top task questions:
    • Your 7600 router crashed after entering the “show mpls forwarding vrf with owner …” command. Determine if there are any workarounds for this known issue.
    • Obtain the latest firmware for your RV082 router compatible with Version 3 hardware.
  2. About 18 test participants are carefully selected based on appropriate profiles.
  3. On an individual basis, using Webex, these people are given the tasks and are asked to try and complete them using their own computer in their own environment to make it as realistic as possible.
  4. The sessions are recorded (audio and video) and the following key metrics are compiled:
    • How many people successfully completed the tasks?
    • Were there any disasters? (Participants think they got the right answer when in fact it was wrong.)
    • How long did it take each participant to attempt each task?
    • What were the major problems that reduced success rate and/or slowed people down?

Scores for each task are derived from these metrics and then synthesized to generate an overall Task Performance Indicator score for the Support Web Site. These new measures which track customer task success have become a key management metric for Cisco.