Category Maturity Scorecard - Release Management FY21-Q1 - Create a Release and update it

When considering the experience our users are trying to create with Release Orchestration capabilities, our focus is to ensure that the job and the context of customers is reflected in our design and engineering efforts, so that we can make decisions and develop capabilities that help customers complete jobs they already want to perform with GitLab.

We use the Category Maturity Scorecard process to grade the maturity of our products based on user performance and feedback.

Maturity evaluation

Previous maturity evaluation

Release Orchestration was made available in GitLab with the implementation of the Releases Page in 11.7. The Maturity plan to bring Release Orchestration to "Minimal" was to expand the Releases API capabilities to include:

Current maturity evaluation

After implementing these feature sets, we saw an increase in the Releases Page on GitLab.com, as well as adoption from our internal teams, such as:

In 12.9, we declared Release Orchestration is Viable as a result of the sustained production usage of Releases. In addition, when we conducted a baseline of "Create a Release Job To Be Done", we learned the following behaviors were expected to reach viability:

We then turned a focus to building in 12.8 and 12.9:

See more on gitlab-com/Product#908 (closed)

Future plans: Next maturity evaluation

Now that those two features are delivered, the top pain points for the non-API users have been resolved and we are expecting the adoption of Releases Page to further increase.

Our process to the Category Maturity Scorecard

The Release Orchestration maturity progress happened in three distinct phases:

  • Phase 1 (12.0 milestone): Validation of the UX Scorecard for Releases (heuristic only)
  • Phase 2 (12.8 milestone): Re-validation of the UX Scorecard for Releases
  • Phase 3 (12.9 milestone): Validation of the solutions proposed by the UX Scorecard for Releases

This document will describe the actions and results from each phase of the process, and how they influence the progression of the Category Maturity level for Release Orchestration.

Identify Jobs customers are trying to get done

Together with Product Management we identified the top 5 tasks in frequency or importance for users of the Release stage group, using our knowledge of the product when lacking research insights.

When asking "is there a job our customers are trying to get done, but can't?", Creating a Release and updating it was the first job identified. We identified the following job statement:

When tracking important deliverables in my project, I want to easily create and manage release entries, so I can provide packaged software, notes, and files for people to use.

CM Scorecard preparations

Defining and recruiting participants

We recruited a total of 6 participants, both internal and external GitLab customers.

Participant profile:

  • Release Managers (enterprise)
  • DevOps Engineer involved in executing releases
  • Automation engineers who are working on releases
  • Users of a project who aren’t necessarily part of the core team - This can be people who use a project's code/binary but are not otherwise involved, and just want to be aware when a new version comes out.

Additionally, Product Management interviewed ~45 customers to talk about Release Management with GitLab:

Interview notes and assets - UX

  • Phase 1: UX Scorecard: Create a Release and update it - Interview notes (#907 (closed)) · Issues · GitLab.org / UXR_Insights
  • Phase 2: &2772 (closed)

Turning JTBD into research scenarios

Scenario

You want to start coordinating a new release for your project. As part of the new release, you need to publish release artifacts and embed checks and steps into the process. Using the <prototype?> show me how you would do this.

Prepare your testing environment

We prepared a test project using gitlab.com (https://gitlab.com/rayanaverissimo/gitlab-design-516). All participants were asked to provide their account handle prior to the interview, and they were invited to the project as developers during the interview.

The scenario requires interacting with the web-based interface for Releases and Tags, and did not require external action from users. Participants should be able to complete everything on their own.

For the second round of interviews, we prepared a prototype of the Releases workflow with improvements and enhancements identified from the first round of interviews (https://rayana412729.invisionapp.com/public/share/6510TVIPSE). Since there is only one main path through the interface to completing the scenario we identified, the prototypes had a meaningful effect on the experience because we could speed up the validation process without implementing changes on code.

Documenting the scenario

The scenario was documented in two different phases:

Writing a testing guide

The UX discussion guide was documented in two different phases:

  • Phase 2: 12.8 milestone - Discussion Guide - Create a release and update it
  • Phase 3: 12.9 milestone - Discussion Guide - Release prototypes

Analysis

Heuristic evaluation / Response rubric

  • Phase 1: How well did the participant complete the scenario?
    • Partially successful
      • The participant completes the workflow but in a manner that doesn’t actually satisfy the conditions of a scenario.
      • Participants were able to create a release only using the API. GitLab provided the option to manage some of the aspects of a Release, but users still had to switch context and could not complete the task only using the interface. Dev centered personas were more successful than non-dev personas.
  • Phase 2: How well did the participant complete the scenario?
    • Successfully
      • Workflow is smooth and painless. Clear path to reach goal. Any extra interactions are very minor and have no meaningful effect on a participants’ progress.
      • Users were able to create and manage a release using the product UI. GitLab offered on the UI the same attributes that a user can manage from the API. Both dev and non dev personas could complete their tasks.

Links

Edited by Rayana Verissimo