KR: Understand the impact of our SUS-related changes more quickly by piloting a process for iterative testing => 83%
Background
In FY22 we strive to improve the areas we identified to have a positive impact on GitLab’s usability and user experience. To allow product teams to continuously listen, test and iterate with end users during development, we want to pilot an iterative test process supporting work on SUS-specific improvement. This process will help understand the impact of our changes more quickly.
The proposed process
We propose the following set of metrics to be collected during usability testing. Having a standardised set of metrics that is continuously used as teams are iterating and testing their solutions, helps to assess impact of changes over time.
ISO's usability definition focuses on three factors: Efficiency, Effectiveness and Satisfaction, which form the basis for the proposed metrics. In addition we include an adjective rating scale that is highly correlated with SUS scores.
| Factor | What’s being measured? | How are we measuring it? | Why are we measuring it? |
|---|---|---|---|
| Effectiveness | Task success/completion rates per task + capture why participants are failing |
We want users to succeed in reaching their goals. Understanding why they fail will help to improve the prototype. | |
| Satisfaction/Efficiency |
Single Ease Question per task + ask participants why they rated it that way |
“Overall, this task was…” - Extremely difficult - Difficult - Neither easy nor difficult - Easy - Extremely easy “Why?” |
It’s one of the questions we use during a CM Scorecard. We are only talking to a small number of participants which makes it important to understand their reasons for giving a rating, especially in situations where a rating is low |
|
Adjective rating scale after all tasks are completed + ask participants why they gave that score |
“Overall, I would rate the user-friendliness of this product as:” - Worst-imaginable - Awful - Poor - OK - Good - Excellent - Best imaginable “Why?” |
It highly correlates to SUS. It’s only one question, but research has shown that the adjectives used in the answer scale correlate to the SUS numerical scores. It’s important to remember that this metric will serve as an indicator for SUS and not replace it. As we talk to a small sample size we can’t just collect the score and need to know their reason for this score. |
Challenges
- Current documentation in the handbook is lacking details around usability testing in general. This needs to be updated as part of this KR.
Open questions
-
How well does the proposed process fit within the workflow of a Product Designer, who will be responsible for most of the work? What are obstacles that might prevent teams from piloting the process?
-
Which templates can be provided in usertesting.com (or other tools) to ease the process?
Checklist
-
Review process proposal with Director of Product Design -
Add documentation about Usability testing to handbook -
Share process with GitLab teams -
Find at least one team that is piloting the process in Q1 -
Collect feedback from teams piloting the process -
Adjust process as needed and update documentation in the handbook