User research works best when you match your participants to the people who will use your designs. It makes sense that teams would try to use the demographics, often compiled by the organization's market research team, as the basis of their recruiting efforts. However, this can be problematic.
To explore why this might not be such a good idea, I recently talked with usability expert, Dana Chisnell. Dana is the co-author of the recently published second edition of the Handbook of Usability Testing, and runs UsabilityWorks, a San Francisco usability research consultancy. Dana's organization recruits hundreds of participants every month for teams all over the world, so she is well familiar with the traps of using demographics. Here's what she had to say about it:
UIE: Recently, we've had a bunch of clients come to our doorsteps thinking they know who they should be recruiting for their usability tests. But what they really have are demographics, such as "70% are males between the ages of 18 and 24." I thought we could talk about why demographics are the wrong way to think about getting test participants.
Do clients come to you with demographics as a description of their ideal usability test participant?
Dana Chisnell: This happens all the time. Especially if marketing or market research is sponsoring the study you're recruiting for, it can be really hard to break out of matching market segments.
Recently, a client came to us (my recruiting consultant, Sandy Olson, and me) wanting to bring people into the lab to compare PC security software. They gave us a screener with percentages. How do you get percentages of individuals?
They wanted 25% of their 24 participants to be between the ages of 30 and 39. They wanted 50% of their participants to be female, 50% to be male. They wanted participants who were professionals, broken down into nine categories. One category was "Banking, investments, and real estate." Another was "Engineering, other." There was also "Education, training, students."
If they've done the research that tells them a quarter of their audience are in their 30s, there is a logic to wanting to recruit participants that way. Why did you feel this approach wasn't going to work for them?
None of those percentages had anything to do with what they wanted participants to do in the study. None of those percentages said anything about how motivated the people we selected might be to do the tasks that were part of the study.
For example, age is not necessarily an indicator of behavior, performance, or expertise — the attributes that make a difference in the results of a usability results. Just because someone is in her 60s, doesn't mean that she's more or less technologically savvy or more or less security conscious than someone in his 30s or 40s. Being 20 doesn't make you an expert computer user. Being 70 doesn't mean that you don't know how to use a computer.
If we wanted to see how different behaviors, performance, and expertise affects how people will use this application, we needed to look beyond age. Professions won't help either. The client gave us a law enforcement category, but otherwise it's unlikely that someone who is a teacher is more likely to be security conscious than someone who is in real estate.
But, in many organizations, the market research groups invest a lot of time and money in developing these demographic descriptions. Isn't using this for recruiting exactly the reason why they were doing this? Shouldn't this be the starting point?
Market research groups generally develop segmentation descriptions to split their market up into sub-markets to address advertising needs. Often, the segments get used to recruiting for focus groups. This is fine if you're doing focus groups to explore market- or advertising-related issues. And, it works well if you're conducting lots of focus groups so you can generalize preferences. However, it's not so good if you want to learn about how well a design works for real people.
At the beginning of a user experience research project, it's okay to talk about what the segments look like. The segments are one view of a user profile; personas are another. But they're based on totally different data. Segments come from self-reported survey data, usually. Personas should be developed after observing users in their own contexts doing their own tasks.