The process of usability testing

Tests of existing products vary greatly based on the goals of the stakeholders, the nature of the product, and the type of users.  In general,  however, I like to follow a basic process for usability testing of an existing system.

1. Discuss the stakeholder's goal

In early discussions, it is important to understand the stakeholder's goal for usability testing. This helps identify the scope of the project. A simple question about an existing feature - such as "Do users understand what this number represents?" - might be answered in the same day. Other questions require longer investigation and preparation. A common impetus for usability research is when Product Management hears vague negative feedback about "feature x", and they want to know how to improve it.

The stakeholder's goal will naturally lead into one or more questions that can be answered with research.

2. Decide on an Approach

Once there is an understanding of the goal and questions to study, the next challenge is to choose the ideal way to answer these questions. Timeliness and access to users are critical here.  At times, full usability sessions are not needed - data trackers like MixPanel or New Relic can answer basic questions about feature use and re-use.

Often, a full usability study is required - usually a user sitting in front of a computer/tablet/smart phone, performing tasks and answering questions about the ease-of-use and satisfaction of a certain feature.  In these situations, users must perform a realistic problem rather than completing basic instructions. Imagine, for example, a study that must identify the problems with editing Terms and Conditions of a contract.  There is a major difference between these usability instructions:

• "Please edit the second paragraph of the terms and conditions for Company X"

• "You are sending a new contract to Company X, who hasn't been paying their bills within the previously contracted timeline. Because of this, finance wants to ensure they prepay before we complete any service for them."

The first instruction focuses on the interface.  It does not give context into why a user must do this.  It is probably giving hints about words used in the interface.  This is unlikely to reveal many usability problems.  The latter instruction is more relevant to the user's goals, and can reveal a disconnect between those goals and product features.

3. Create a research plan

A research plan concisely documents the decisions made so far, as well as the intended set up of the usability sessions (if needed).  The example here is based roughly on a traditional layout of a scientific journal article.  The goal is to keep this plan to a single page - any more, and the plan is probably too complicated for a single usability study.

This plan should be reviewed with stakeholders, in order to confirm the questions to be answered, and how researchers intend to find those answers.

Once the research plan is approved and the testing has finished, I like to add results & conclusions to this document, as well as an executive summary.

4. Create testing environments

Some testing requires nothing more than the user's own instance of the product, but more often they require unique environments with constructed data and early design prototypes.  When trying to confirm results between users, it's important that each user interacts with the same instance of the product.  This requires some discipline on the part of the researcher, to ensure each user has the same experience during feature testing.

5. Test & analyze

When testing features in a usability test, data should come from multiple places, such as the following:

• Speed and accuracy while finishing the task

• Attempted paths to task completion

• User-reported feedback, including Likert scale questions and comments

• Facial expressions during the task

Invite stakeholders and development teams to attend sessions, if possible.  Videos of participants and content can be a good alternative for those who cannot attend.  The Usability Testing Dashboard I show here is useful for bringing the feature team's attention to current testing.  I've adapted this testing dashboard from David Travis, with changes based on my experience - offer short, concise information and include images of what I'll be testing, when relevant.

6. Report and Present findings

Researchers should create a quick report that others can review on their own time.  A condensed version of the research plan should be the first part of this report, reviewing the goal, questions to be answered, and method used.

Many stakeholders prefer a quick presentation of the results, which can create a healthy discussion during and after the presentation. Another great way to illustrate findings is to have stakeholders attend usability sessions. This allows them to see and hear the feedback directly from the user. Videos of participants and content can be a good alternative if stakeholders cannot attend.

7. Follow-up

Just like Agile development is a cyclic process of a continuously improving product, usability testing can show improvements from redesign and development.  A former colleague of mine, Arno Huang, was able to show the continuous improvement of the product after multiple cycles of redesign and new features.  He found that user acceptance, satisfaction, and task completion speed all increased with the continued development of a feature.