3 Common Misconceptions about User Testing

Context Instead of Personas - From Inside Intercom

Meanwhile, a lot of company’s building (digital) products have embraced (or at least accepted) the positive impact user testing can have in the product development process. But just because you somehow involve users into your operations, doesn’t mean you’re doing it right.
Personally, I had the luck to get great teaching into the art of user testing from my former colleagues Britta and Anne. And while you can and have to divide user testing into its separate domains (research, deep interviewing, usability testing), I want to point out three widespread misconceptions which I stumble upon in early discussions.

Focus Groups are the shit!

Nope, they’re not. Focus groups are one of the worst ways to get feedback on your product or getting to real user struggles (right behind building assumptions just on your behavior). When it comes to pointing out failures of a product, people are herd animals. They’ll follow the most common opinion in the room and will be too embarrassed to be the first ones acknowledging a mistake or flaw.
Only personal situations and a comfy and trustworthy environment will bring you closer to see the real behavior of people and hear about their real struggles.

If you want to have the room full of like-minded customers of yours as part of your product creation process, I instead recommend the format of Co-Creation Sessions.

Personas are a great way to figure out my user’s needs

While personas are already quite an improvement for user-centered products, I still think they have a huge flaw: They focus on the wrong things and distract you!

Building upon what the smart guys behind Inside Intercom have written about why Job Stories beat the format of User Stories, I want to apply to the personas framework. Conducting the cornerstones of a ‘traditional’ persona (age, hobbies, family status, hopes & dreams) takes way too much time (or is purely based on imaginative stereotypes) and doesn’t give you any solid entry points for figuring out around which problem your product should solve.
Instead, you start chasing constructed touch points in a user journey far off a users core problem (you’re able to solve). Alternatively, you should almost entirely focus on the context, and the central issue which will inevitably lead to a sure ‘job‘ people hire a product/workflow for. After figuring those two things out, the demographic data of your target group becomes highly irrelevant.

If you want to apply a Persona-style approach at the beginning of your product discovery anyways, I’m a firm advocate of Proto-Personas. This format will keep your efforts focussed on demographics reduced to a minimum and will include some reality checks.

We’ll just ask them what they want and how much they’re willing to pay for it

While I can’t stand the iPhone comparison when it comes to user research (‘Steve Jobs never asked anybody, whether someone wants an iPhone), there’s a certain point in it: Users are not able to articulate the right solution to their problems. This is why you are a tasked with building products which serve their needs. But to get there, you have to entirely focus on issues and not potential solutions, when talking to users before even sketching a first wire frame.

But to get there, you have to solely focus on problems and not possible solutions, when talking to users before even drawing a first wire frame. Don’t ask them what they want (also if it itches you to shout this question at them after they’ve neglected 6 of your fancy prototypes) and instead focus on what makes them happy or grumpy.
And while you’re at least able to derive some valid qualitative data from talking to users about problems, you won’t be able to when it comes to monetization. Sure, you’ll get close to figuring out a person’s most important product attributes which make them paying (e.g., quantity over quality or cross-device availability). But whether they’ll pay in the very moment for your product needs to be determined outside of that interview room (I’ll talk about quantitative validation at a later point).

In addition: Here are some Customer Interview Questions You Should Never Ask

  1. “What do you think of our product?” – Alternative: “What’s your workflow like when you use our product?”
  2. “What features should we add?” – Alternative: “What tools do you use alongside our product?”
  3. “How often do you do X?” – Alternative: “When was the last time you did X?”
  4. “Would you use/buy this?” – Alternative: There’s no good way to ask this.

Which misconceptions and mistakes about user testing are you giving you a hard time to get out of people’s heads? Check out this great article at Inside Intercom when you’re looking for more mistakes.