From a measurement perspective, even this was a bit of old news. There has always been multiple ways to gauge the effectiveness of online initiatives; user experience testing, surveys, focus groups, market research are the primary tools that come to mind. Add in social media measurement tools and mobile analytics and the data starts to pile up pretty quick.
How do you know when to use each of these tools and techniques?
Over the years, I’ve grappled with this challenge and during a recent engagement at the National Institutes of Health I refined and introduced a methodology that I like to call “triangulation.”In the “real” world, triangulation is a term used in surveying and social sciences. Read the definition provided in the American Heritage Dictionary and I think you’ll agree that we can apply this to the use of disparate data sources:
A surveying technique in which a region is divided into a series of triangular elements based on a line of known length so that accurate measurements of distances and directions may be made by the application of trigonometry.”
Social science researchers have adopted this term to describe methods for validating the results of their work. This is precisely how we can use this technique…using different data sources to both inform and cross check.
Figure 1. Triangulation is a method for comparing several Web analytics data sets.
Triangulation and the Website Development Process
Before we discuss how the different types of measurement work together, we need to describe how digital channel project development occurs. Basically, there are two types of projects:
- content, with text pages with links; this can include campaigns, such as email blasts
- applications, with interactive elements, such as forms, shopping carts, search engines and data-selection wizards
The only real difference between these two is in size: some projects are small, such as creating a micro site, and some are large, such doing a major site redesign. However, the steps are the same for all projects. In Figure 2, we illustrate five basic steps in Web development projects and how to incorporate site measurement into them.
Figure 2. Digital development process and relevant site-measurement options.
Now, let’s see how we can use the right measurement technique at the right point in the development process to inform what we’re doing.
Step 1: Requirements. Requirements state what the digital project will be and what elements it will include. Developing requirements may include preparing use-case scenarios to suggest how visitors will interact with the site.
How to triangulate site-measurement methods for requirements:
- Use analytics for historical data to understand how visitor segments currently use content and applications to get ideas about the audience’s interests.
- Use focus groups, surveys, email, and call-center contacts to stimulate ideas for brainstorming about development.
- Use market research to understand how your target audiences are using similar websites.
We recommend that by the time the design phase is complete, the development team has prepared research plans that consider all the elements to be used for measurement and testing in conjunction with the new initiative. The team should emphasize mapping how the qualitative and quantitative data would be used in tandem.
For example, if customer-satisfaction surveys are to be used to determine whether visitors are satisfied with the workflow of a new wizard application, a corresponding Web analytics funnel analysis should be conducted to validate, correlate and expand upon the surveys’ findings. A funnel analysis focuses on the effectiveness of the specific path you’ve laid out for visitors to take to complete a task. The analysis will show where on that path visitors exited.
Knowing what your visitors think, say, and care about is the single most important data asset you can have. Many of my clients are becoming much more aggressive about collecting, consolidating, analyzing and using customer-sentiment data. This is because everything customers tell us as they navigate through a site can help achieve our online business goals. From online surveys, comment cards and call centers to focus groups and social media, organizations have access to enormous amounts of customer feedback and opinions. It's challenging to capture and use the data, and it requires new systems and analysis techniques that are different from traditional Web analytics.
We expect further challenges to materialize with the advent of social media tracking. However, the same practice of goal setting and clarifying how the data correlate with other data streams should help decrease the confusion about what source to believe and how much credence to give one source versus another.
Step 2: Design. Once the requirements are defined, the information architecture and page design can begin. Depicting initial designs graphically may help refine and focus the requirements.
How to triangulate site-measurement methods for design:
- Use Web analytics for historical data to understand any site-navigation successes and challenges experienced by different segments of visitors.
- Use website performance monitoring reports to avoid performance issues that can arise with large graphics files.
Step 3: Development. Once the design has been finalized, the development work can begin to include creating functional specifications and coding.
How to triangulate site-measurement methods for development:
- Use website monitoring reports to avoid performance issues that can arise with densely coded pages.
Step 4: Testing. Conduct all quality-assurance tasks related to development.
Step 5: Launch. When the content or applications “go live” you can collect data on their effectiveness.
This is also when you begin to evaluate how to improve the site or application and how to enhance strategies for driving more traffic to the site.
How to triangulate site-measurement methods for the launch:
- Use analytics to compare new content or applications to your business goals or navigation ease and campaign effectiveness.
- Use customer-satisfaction surveys to assess ease of use and value.
- Use post-use surveys to assess the potential behavioral impacts of new content or applications.
- Use market research to evaluate whether new content has changed the positioning of your site or applications relative to similar or competing sites or applications.
A Closer Look at Measurement Techniques
Let’s start off with basic definitions of quantitative and qualitative tools and methods:
- Quantitative Tools and Methods
- Record performance data generated by the operation of a fixed, mobile or social presence
- Provide information about WHAT actions, events and trends are happening.
- Qualitative Tools and Methods
- Record usage data generated by current and potential visitors.
- Provide information about WHY events and trends are happening.
Here’s a list of the quantitative and qualitative methods and tools used for analyses.
Quantitative Analysis Tools and Methods
1. Web and Mobile: Analytics software is necessary for collecting information on
- visitor activity
- content and function usage
- site promotion and marketing
- task completion
- site usability
- external search
- Internal search
2. Social: Social media has its own set of metrics that are similar but different than fixed web. We’ve listed most common types below:
- Traffic (visits, impressions/views)
- Interactions (likes, comments, posts, tweets, impressions)
- Channel ad campaign cost
- Share of Voice
- Value (or other research based metrics)
- Referred traffic from social to web or web to social
- Referred conversions (or other online successes) from social to web
3. Site Quality and Performance: Website monitoring is necessary for keeping track of Web server performance, such as server availability and response time.
4. Market Research: Internet industry or market research and business intelligence companies recruit users and use Internet service provider (ISP) data to define their Internet and Web users’ demographic and behavioral characteristics. This provides a clearer understanding of who is using your site, as well as your competitors’ sites. Market research organizations include Experian Hitwise, ComScore, Alexa.com and Google.
Qualitative Analysis Methods and Tools
1. Usability Measures -- User-Experience and Usability Testing: This kind of testing is used to evaluate website design for usability (for example, ease of finding information, reading a site, using site navigation, following workflow processes, filling out forms and using search engines, multimedia and graphics).
User testing is necessary for understanding
- user reactions to proposed site designs and navigations
- why users leave the site
- why users like or dislike the site
- why it’s easy or difficult for users to find what they are looking for
- whether the site is intuitive enough for visitors to use without assistance
- whether site instructions are easy to understand or lead to frustration.
2. Customer Sentiment -- Email and Call-Center Contact: Direct contact and feedback from site users can raise red flags about issues with site content, function, design and navigation that need to be addressed. For every person who makes a comment, there are many others who have the same issues but do not comment.
3. Customer-Sentiment Online Information Gathering -- Surveys: Customer-satisfaction surveys that deploy during a visit may be for understanding visitors’ points of view as they relate directly to navigation or tasks they are trying to perform. Direct online surveys sent to visitors after use of the website provide insight into whether using the website elicited a change in behavior based on information received or on a specific task completed.
4. Customer Sentiment -- Focus Groups and Interviews: Focus groups and interviews can provide information that you can use for improving and planning outreach campaigns, site content, and site functions.
Digital analytics and the other measurement methods should all be used together through triangulation. We believe that the concept of triangulation makes more sense when it’s viewed within the context of the Web development lifecycle. Understanding the role and value of each type of measurement within the lifecycle helps determine how the information from each measurement method informs the lifecycle activity.
Title image courtesy of Flegere (Shutterstock)
Editor's Note: To read more by Phil see Web Analytics KPIs = BS.