KR: Identify top-10 usability issues in our product => 71%, 5/7
Identifying the Top 10 usability issues
To improve the SUS, teams must have a high degree of confidence around knowing if the usability issues they're going after are the ones that matter most. This KR will detail out a repeatable process that can be used to obtain the top-10 usability issues to provide teams with that level of confidence.
There are potentially many approaches to take in identifying which usability issues to focus on. Since the goal is to improve the SUS, we will take the following approach:
Step 1: Dig into the data
Researchers will start by looking across our data sources to identify existing usability issues. These all have the potential to be fruitful in identifying existing usability issues, without a significant amount of effort.
- Why this approach?
- This holistic approach allows the team to look under every stone for known usability issues that are likely to be impacting the SUS. Known data sources:
Completed? | Owner | Data Source | Link to issue with findings |
---|---|---|---|
Yes | @katokpara | Verbatim/insights from surveys (ex: SUS, NPS, etc); looking for usability themes | |
Yes | @katokpara | Analyze Eric Brinkman's tweet: https://twitter.com/ecbrinkman/status/1297906656597151750 - Ping Eric to see if he coded those responses. Look through those to identify themes related to usability. | |
Yes | @akiripolsky | Look into the 'Frequently duplicated' label for issues that may be usability-related | |
Yes | @asmolinski2 | Analyze UX Scorecard findings to look for usability themes Open issues - Closed issues | |
Yes | @loriewhitaker | Review CM Scorecard findings to look for usability themes that impact key workflows gitlab-org/ux-research#1193 | |
Yes | @alasch | Look through Dovetail & GitLab to identify usability issues that are unresolved for at least 2 quarters | |
Yes | @alasch | Look through studies in the UXR insights repository to identify usability issues that are unresolved for at least 2 quarters | |
Yes | @jeffcrow | User requested issues (how much IACV/how many paid users are represented by accounts requesting features/bug fixes) | |
Yes | @tlavi @akiripolsky | Most popular open issues in the GitLab project (by upvotes) |
- (This will in a large list of usability issues to start with)
Step 2: Develop a matrix
The list of items from Step 1 will most likely be unwieldily and difficult to parse, so a step is needed to make that data more efficient to use.
-
Why this approach?
- Developing a matrix allows us to understand if we're getting signal on the same usability issue from multiple data sources. It also helps us consolidate the list more efficiently, which makes it easier to work within and to reference back to the data source, if needed.
-
Develop a matrix to identify which data source the usability issue was derived from -
(This list will stay the same size, but organized now)
Step 3: Assign priority
Determine which of those usability issues impact users the most. This will be done by looking for a relationship between frequency of encounters and longevity of the usability issue. Additionally, we will prioritize by the Product Usage Driver Score.
-
Why this approach?
- By prioritizing the usability issues, that helps us reduce the list and focus on what's impacting the most users. Addressing those first is likely to have an impact on the SUS.
-
Review the 'most visited pages' to look for a relationship in high/low encounters - Taurie has this list: gitlab-org&4704 -
Search on GitLab to determine the longevity of the usability issues, if they're already documented -
Apply our Product Usage Driver Score to the usability issue -
(The list will be smaller after this cut)
Step 4: Determine severity
Not all usability issues are equal in severity; some are minor and some are catastrophic. The thinking is that the major issues are more likely to be impacting SUS. To determine severity, we’ll leverage existing measures, also listed below.
- Why this approach?
- We want to start by focusing on the issues that are most severe.
- The severity of a usability problem is a combination of three factors:
- The frequency with which the problem occurs: Is it common or rare?
- The impact of the problem if it occurs: Will it be easy or difficult for the users to overcome?
- The persistence of the problem: Is it a one-time problem that users can overcome once they know about it or will users repeatedly be bothered by the problem?
- Finally, of course, one needs to assess the market impact of the problem since certain usability problems can have a devastating effect on the popularity of a product, even if they are "objectively" quite easy to overcome. Even though severity has several components, it is common to combine all aspects of severity in a single severity rating as an overall assessment of each usability problem in order to facilitate prioritizing and decision-making.
-
The following 1 to 4 rating scale can be used to rate the severity of usability problems. Note that this severity model is based on the existing GitLab severity model:
- 1 = Blocker - Usability catastrophe: imperative to fix this before product can be released. Users are blocked (or so confused that they believe they are blocked), and are likely to ask for support.
- 2 = Critical - Users are significantly delayed by the available workaround.
- 3 = Major - Users are self sufficient in completing the task with the workaround, but may be somewhat delayed.
- 4 = Low - Usability isn't ideal or there is a small cosmetic issue.
-
Sign-off on the list by stakeholders -
(After this cut, there will be a Top 10 list of usability issues)
Step 5: Conduct usability studies (not done; deferred to future quarters)
The Research team will divide and conquer to conduct usability studies on the Top 10 list of usability issues.
- Why this approach? This will be done for three reasons: 1) To understand why there are usability issues, 2) to provide more insight into how to correctly address them, and 3) to measure against once iterations are made.
Step 6: Deliver the Top 10 list
A thorough report will be generated, with accompanying video, to detail out the Top 10 usability issues.
- Why this approach?
- This aligns everyone involved with the effort and provides very clear guidance on what the issues are and suggestions on how to address them.
Step 7: Create Actionable Insights (not done; deferred to future quarters)
Actionable Insight issues will be created as part of the deliverable.
- Why this approach?
- This will be done to accurately define and track the usability issues for teams who are working on addressing them.
Checklist
-
Step 1: Dig into the data -
Step 2: Develop a matrix -
Step 3: Assign priority -
Step 4: Determine severity -
Step 5: Conduct usability studies -
Step 6: Deliver the Top 10 list: https://docs.google.com/presentation/d/1kMZcRnOjuQ0K3TtT1YxedhFb5-Ej_cr2mXxeeJ9sbZ0/edit#slide=id.g469c9b8c47_0_0 -
Step 7: Create Actionable Insights
Open questions
- Q: Should certain stage areas be prioritized and/or deprioritized? If so, which ones?
- A: Yes. See the Product Usage Driver Score.
- Q: What if a stage area doesn’t have their JTBD established?
- A: We still move ahead without it. Usability issues associated with a state are without JTBD may have surfaced up in the other data sources.
- Q: What if an experience spans multiple stage areas? It may not appear in an existing JTBD since those are focused at the stage group level.
- A: This is a non-issue since we're not taking a stage area specific approach.
- Q: How do we know which experiences impact the most number of users?
- A: See Step 2.
Stretch goal
- Publicize the top issues being working on in the handbook, for transparency, including updates
END of Q4 update and reflection
The original approach for this KR (still documented above) shifted based on feedback from the Product team. As a result, two items could not be completed:
-
Step 5: Conduct usability studies - Instead of these taking place in Q4, they will take place ad-hoc and as needed, at the stage level over future quarters. Since each quarter will focus on a single theme, usability studies will also focus on that single theme. Overall, this is a more targeted approach that keeps Research in alignment with Product.
-
Step 7: Create Actionable Insights - Since usability studies weren't done in Q4, actionable insights weren't created. It's expected that actionable insights will be created as usability studies are conducted in future quarters.