Mobile Usability Testing for 2010 and Beyond

5 minute read
Jason Munson avatar

As the mobile channel’s audience has progressed from early adopters to the mainstream, many businesses are now providing mobile experiences for their users. The shifting momentum from desktop experiences to mobile has forced many online managers to stretch themselves into understanding this additional channel’s nuances and how to design mobile optimized user experiences (UX). Here's a look at how Usability Testing can help and how it's different from testing for the desktop.

Once best practices knowledge is leveraged to design your first mobile website, every Customer Experience (CX) obsessed professional will want to test, learn and refine. Analytics can help identify the area(s) where Usability Testing is needed, which is a familiar concept and process for most. 

Similarly, most CX and UX focused professionals have a clear notion of what a Usability Test for a desktop website looks, sounds and feels like. Whether the testing is conducted in the lab, field or via remote desktop connection, the stakeholders have a good sense of what to look for and how to make the needed improvements. 

(Also by Jason Munson: Enterprise 2.0: Increase Productivity and Extend Reach via Mobile)

There are similarities in measuring and testing experiences across the desktop and mobile channels, but there are also some new perspectives to be considered. In today’s post I will touch on two of the big decision points that you will need to consider when you tackle mobile usability testing: Identifying the Devices you will test, and the Testing Methods and Environment.

Identifying the Devices

When running a usability test for mobile devices, one of the first things to consider should be the devices you are going to test. This can be a significant challenge, as the device pool is constantly evolving with newer-cooler-sleeker products hitting the market on a weekly basis.

The obvious first step is to know your audience, and consult your analytics to get a sense of the devices that are being used to access your existing web property. Even if your project is the first mobile channel experience your brand is releasing to market, knowing who’s tried to interact with your site via mobile is certainly important.

Once you know what the device pool looks like, you’ll either need to make a determination on the specific devices you want to test (Apple iPhone 4, Motorola Droid 2, Dell Streak, etc.) and consider things like screen size, screen layout and interaction style... or ... define and choose device categories for testing.

Testing Device Categories

Testing device categories is a good option for companies that have decided to cater the experience to a more granular degree than “one size fits-all” (a.k.a the Lowest Common Denominator) but are not optimizing on a device specific level. If this sounds like a sensible approach, you’ll want to make sure you make this decision starting at the design phase, to achieve optimal results.

Some examples of possible device categories are:

Learning Opportunities

  • Touch-Screen Mobile Phones (Apple iPhones, HTC Nexus One, HTC EVO, etc.)
  • Non Touch-Screen Mobile Phones (RIM Blackberry Bold, Motorola Droid 2, etc.)
  • Quick Messaging Devices (LG Xenon, Samsung Intensity, etc.)
  • Tablets (Apple iPad, Samsung Galaxy, Dell Streak, etc.)

Testing Methods and Environment

The existence of statistically meaningful differences in test results conducted on mobile user interfaces under different environments remains a topic of debate for many.  

The primary methods for testing are:

  • Using mobile device emulators on a computer in the lab
  • Using mobile device emulators on a computer via remote desktop connection
  • Using a mobile device in the lab
  • Using a mobile device linked to a camera in the field

From my point of view, you should consider what you are trying to learn from the test to help your decision on this front. If you are looking for simple insights on cosmetic and navigational changes to the mobile experience on a device of choice, then a lab setting is likely sufficient. You can start by testing early concepts with mobile sized paper prototype, follow-up with an emulator and wrap with a lab based mobile testing session. 

If on the other hand, you are trying to learn about the complete Customer Experience, then a field test is the better (I might argue the only) option. Multi-tasking, lighting, volume controls, background noise and the user’s movement pattern during interaction are all factors that should be considered during design and testing. Additional factors are the points of interaction with the environment around the user that the new mobile products are capable of.

Consider some of the products we are seeing come to market in the mCommerce space recently. There are game-changing mobile products out there now. You can walk around one major retailer while comparison shopping at another via your mobile device (and buying for immediate in-store pickup), you can scan 2D barcodes or run searches for consumer reviews of the product by taking a picture with your mobile device, and receive contextual SMS promotions when you walk by the new connected TV displays thanks to the Location Based Services (LBS) functionality.

Put the Customer Experience First & Be Bold  

These new mobile experiences are simply a glimpse of the innovation that the mobile channel will be introducing in the next few years. Winning brands are grabbing market-share through differentiating mobile experiences that are based on thoughtful designs that put Customer Experience above all else.  My advice to brands that are just flirting with this channel is to be bold with your mobile investments -- not timid -- or you may be left behind.

You might also be interested in Why Mobile Websites Are Better than Mobile Applications.