Qualitative Usability Tests

Before we talk about Qualitative Usability Tests

We first need to talk about what a Usability Test is in general.

A Usability Test is a method used to evaluate how usable or intuitive your product is. This is done by observing and analysing representative users as they perform specific tasks in your product. Usability Tests allow you to identify any usability problems, collect qualitative and quantitative data, and determine your users’ satisfaction with your product or proposed designs early on in the design cycle.

There are two types of Usability Tests; Quantitative Usability Tests and Qualitative Usability Tests. Quantitative usability testing is focused on collecting UX metrics (like time on task or task success) through controlled, specific tasks. A qualitative usability test has more open-ended tasks and prioritizes observations, like identifying usability issues or user insights.

For this method, we are going to focus on qualitative usability tests, but you can find information on quantitative usability tests here.

What is a Qualitative Usability Test?

A Qualitative Usability Test is a method that prioritizes observation when participants perform open-ended tasks in a system as you collect feedback on usability issues or user insights.

Qualitative Usability Tests are a type of Formative User Research, which basically means that the results from these tests inform how the design will evolve during the design process, however, they can also be performed on your existing product in order to inform the next round of design changes.

Since the goal of this method is to guide your designs, qualitative usability tests are effective even with smaller sample sizes as you can get a lot of contextual feedback and quickly iterate to achieve your goals.

Some cons of qualitative usability tests are that they can be time-consuming, recruiting can be a challenge, and they can take a long time to analyse. Remote moderated testing can help make scheduling easier, and there are paid services out there to help with that, however, they can be quite expensive.

What do you need for a Qualitative Usability Test?

TIME

  • A few days for scheduling participants and setting up the tasks
  • 15 minutes to 1 hour to conduct the test

MATERIALS

  • A prototype or existing product
  • Recording equipment
  • Something to take notes on

How do you conduct a Qualitative Usability Test?

Step 1: Decide what you want to test and why

As mentioned above, qualitative usability tests can be run on early designs or your finished product. You just need to know what you want to test and why. For example, are you looking to evaluate certain areas of your existing product to uncover new potential features or fixes? Or are you looking to evaluate your early designs to see if you’re on track and to guide the next round of quick iterations?

Step 2: Decide if you want moderated or unmoderated

Your next decision is to decide whether it will be moderated or unmoderated. A moderated test involves the active participation of a trained facilitator, while an unmoderated test is completed by test participants in their own environment without a facilitator present.

Unmoderated tests are faster, and allow for greater sample sizes in a short amount of time, but there are several cons; you can’t ask follow ups, the data has higher variability, and there is a risk of “Cheater” participants (which are participants that are only in it for compensation and do not provide substantial feedback).

Step 3: Create your test plan and tasks

Next, you need to create your test script and tasks. Come up with as many tasks that you want feedback on and that will fit within your timeframe. Your tasks here should be open-ended and the flows your participants interact with should support the entire task and any possible edge cases.

Some examples of qualitative tasks would be:

  • Find a pair of runners that you like and pay for them using our site
  • Find a way to get help on the site
  • Use this dashboard to analyze your data

Step 4: Schedule and recruit participants

Once you know what you want to test, you will want to find participants. Ideally they are representative of your target audience and have the right amount of experience for the flows you are testing. For example, if you’re testing for feedback on a new sign-up flow, you may want participants that have never used your product before. If you’re testing an advanced feature, you may need participants with a lot of experience with your product.

Qualitative usability tests are effective with small sample sizes (around 5 participants per round). The logic behind this is that after around 5 participants, you will start getting too much repetitive feedback, and it will no longer be worth your time and effort. Instead, you’re encouraged to make changes to your designs, then do another round of tests. Repeat this process until you feel you’ve reached your goal (or until you run out of time).

This makes scheduling easy, since you only need 5 per round, but depending on how many rounds you do, scheduling could become difficult.

Step 5: Have them go through the tasks and note your observations

Now you can have them go through the tasks while you record and document it. During the tasks, ask them to “think out loud”. Encouraging participants to talk through what they are looking for and why will unlock deeper insights into what they want, which will give you plenty of ideas on how to address it. If they start to get quiet, you can always ask “what are you thinking now?”.

Step 6: Ask questions

In between each task, or once all tasks are complete, you can bring in some other research methods, like Quantitative Surveys and Qualitative Surveys, to get some additional feedback on the specific task or overall flow. You can even interview them with another set of questions, however this can only be done in moderated tests.

When listening to their answers, it’s important to prioritize their actions in the tasks, rather than what they said in their responses. For example, they may tell you that they found the task easy, but in reality, they may have struggled for a long while at various points in the task.

Step 7: Analyze and report your findings

Pull out the main takeaways from each task then cross reference those takeaways with the results from the other participants. This will surface any patterns or trends in the feedback and possibly point to deeper insights. It will also allow you to make design proposals based off of user data, instead of just your opinions.

Once you have this all in an easy-to-digest report, share your findings with your team and stakeholders. Make sure to include the main insights at the top, alongside any proposals you have on how to improve the designs before your next round of Usability Tests, or before you hand the designs over to the dev team.

Tips for a great Qualitative Usability Test

  • If they ask you questions or for help during the tasks, try to avoid answering them until the end of the test. You can simply say “I’m sorry, but I can’t answer that right now, as I would like to see what you would do if I wasn’t here to help you, but I would be more than happy to answer any questions you have at the end of the test.”
  • Do a rehearsal Usability Test before you do it with an actual user. This will let you test all equipment, and to see how if your script or tasks need adjusting to better fit within your timeframe. For this rehearsal, you can use anyone from your company.
  • You can still collect metrics (e.g. success/failure) in a qualitative usability test, but that doesn’t make it a quantitative usability test, typically because the sample size is too small and the tasks are too open-ended.
  • Always remember that it’s more important to observe what they do, rather than what they say in response to your questions. Usability tests are about gathering observations over opinions, and what they say may drastically contradict what they actually did in the test.
  • If you don’t have time to design for each possible edge case, you may need a generic “under construction” page to present to the user, then you can steer them back on track if they hit that screen. Just make sure to note anytime that it happened, and if you can, ask them questions on why they chose to go there.
  • Some good questions to encourage “thinking out loud” while they perform their tasks are:
  • What are you thinking now?
  • What are you trying to do?
  • Why is that important to you?
  • When analyzing the feedback, you should use your best judgement when deciding what to surface to your team. There may only be 1 participant that found a specific usability issue, but if it’s a big one that you feel should be addressed, you may want to include in your top takeaways.

More resources for Qualitative Usability Tests