FY21-Q1 UX Quality Product OKR: Uncover unknown areas of low category maturity so that improvements can be prioritized => 72%

KEY RESULTS: User validate category maturity ratings for all categories that moved to the next rating during Q4 FY20 (except Minimal, which is self assigned).

  • KR1: In the handbook, document a repeatable process for (1) synthesizing UX Scorecards for Category Maturity research to create validated ratings and (2) creating resulting recommendations issues. Get the new process approved by Product Ops. => 100%
  • KR2: Interview at least 30 users (5+ users per 6 categories) to evaluate the experience of 6 categories that changed maturity levels in Q4 FY20 and Q1 FY21. => 4 of 6 Scorecards completed = 67%
Category Previous Maturity Validated Maturity Notes Status
Release Orchestration Minimal Viable Product #908 , gitlab-org/gitlab-design#1115 (closed) ✅
Package Registry Minimal Viable Moving to future milestone. Category Maturity Scorecard process uncovered that the JTBD as defined was incorrect. New JTBD now defined for future validation. Not completed
License Compliance Minimal Viable Cateogry Maturity Scorecard not prioritized this quarter Not completed
Serverless Minimal Minimal No longer a focus for Configure stage Not completed
Logging Minimal Viable Category Maturity Scorecard work uncovered a critical usability problem that we have since fixed. With this change, we believe that the category is Viable. gitlab-org/gitlab-design#1120 (comment 332903808) ✅
Error Tracking Minimal Viable gitlab-org/ux-research#651 (closed) ✅
Roadmaps Minimal Minimal Category Maturity Scorecard uncovered that the category should not move up to Viable ✅

Product OKR for reference: Product#550 (closed)

Contextual information ### Goals

Short term:

Long term:

  • Iteratively validate scores for all categories over the next few quarters to reflect category maturity levels that align with user perception
  • For every category that proposes moving up a level, ensure that category maturity is validated

Proposal

Identify Primary and Secondary JTBDs

Each Product Designer works with their Product Manager to identify Primary and Secondary JTBDs within the category

  • Can be a previous or new UX Scorecard
  • Must include at minimum 5 participants in a scenario-based observation study (the scenario is the JTBD)
  • Each participant is observed and at the end of the session asked to provide an Experience Grade (based on the Grading rubric)
  • Participant-provided Experience Grades are averaged to determine the UX Scorecard grade

Review results

Product Designer and Product Manager review the results to determine if UX Scorecard grades align as follows:

  • Loveable
    • Current Definition: Provides an elevated user experience that customers love.
    • Proposed Score: Received an A for the primary JTBD and Bs or higher for secondary JTBD in user-validated UX Scorecards.
  • Complete
    • Current Definition: Contains a competitive feature set sufficient to displace other single-purpose DevOps tools.
    • Proposed Score: Received a B or higher for all primary and secondary JTBD in user-validated UX Scorecards.
  • Viable
    • Current Definition: Can be used by users/customers to solve real problems.
    • Proposed Score: Received a C or higher for JTBD in user-validated UX Scorecards.
  • Minimal
    • Current Definition: Provides a minimal foundation so people can see where we're going and to validate customer need.
    • Proposed Score: Assigned by Product Manager and Product Designer based on Solution Validation activities.

Compare with current Maturity Levels

Complete top JTBDs, average the scores, and adjust current Maturity Levels, if needed.

Good

  • We were able to validate the maturity level of 4 categories with customers
  • One category that was planned to move from Minimal to Viable was held at Minimal as a direct result of this process

Bad

  • We did not prioritize completing the process for two of the categories

Try

  • Simplify the Category Maturity Scorecard process
Edited by Christie Lenneville