Skip to content

UX Scorecard - Verify:Pipeline Authoring FY25-Q2

Related OKR for FY25-Q2: https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/6987+

UX Scorecard Checklist

Learn more about UX Scorecards

Click to expand
  1. Add this issue to the stage group epic for the corresponding UX scorecards. Verify that the "UX scorecard" label is applied.
  2. Work with your PM to identify a top Job to be Done (JTBD). All GitLab JTBD can be found in the jobs-to-be-done.yml file. If creating a new job, write it using the JTBD format: When [situation], I want to [job], so I can [expected outcome]. Review with your manager to ensure your JTBD is written at the appropriate level. Remember, a JTBD is not a user story, it should not directly reference a solution and should be tool agnostic.
  3. Make note of which personas might be performing the job, and link to them from this issue's description. Keeping personas in mind allows us to make the best decisions to address specific problems and pain points. Note: Do not include a persona in your JTBD format, as multiple types of users may complete the same job.
  4. If your JTBD spans more than one stage group, that’s great! Review your JTBD with a designer from that stage group for accuracy.
  5. Add any new JTBD to the SSOT jobs-to-be-done.yml file
  6. Consider whether you need to include additional scenarios related to onboarding.
  7. Select the appropriate scorecard approach and evaluate the current experience.
  8. Use the Grading Rubric to provide an overall measurement that becomes the Benchmark score for the experience (one grade per JTBD), and add it to this issue's description. Document the score in the UX Scorecard Spreadsheet.
  9. Once testing is complete, create a walkthrough video that documents what you experienced/witnessed within the existing experience. Begin the video with a contextual introduction including: your role, stage group, specify how you acquired the data (ex: internal or external users, or self-heuristic evaluation), and a short introduction to your JTBD and purpose of the UX scorecard. This is not a "how to" video, but instead should help build empathy for users by clearly showing areas of potential frustration and confusion. (You can point out where the experience is positive, too.) At the end of the video, make sure to include narration of the Benchmark Score. Examples here and here.
    • If you're re-scoring the experience, walkthrough the entire flow again. For narration, you can highlight the recent improvements but still call out any areas that could still use some tweaking (in the next round of iterations, if applicable). The re-score video, in theory, should be shorter since we've hopefully eliminated a few bumps in the user flow.
    • The walkthrough video shouldn't take you long to create. Don't worry about it being polished or perfect, it's more important to be informative.
  10. Tag PM and UX DRIs for this JTBD in this issue to share findings.
  11. Post your video to the GitLab Unfiltered YouTube channel, and link to it from this issue's description.
  12. Link to your video in the Engineering Week in Review and Product Division Monthly Sync.
  13. Create a new Dovetail project using the UX scorecard template. Use insights to document any observations or findings that came out of this scorecard. You can use your experience map or video summary to help you curate those. It is important to add insights into Dovetail so they can be shared and accessed by all groups, and used to document cross-stage findings. You can also add any supporting material in Data, such as an exported Mural experience map, but it is not required. Example here.
  14. Create a recommendation issue for this JTBD and add it to the same stage group epic as this issue. Also add a link to your recommendation issue to this issue.

Formative evaluation session details

Click to expand

Recruitment

3-5 GitLab team members who have experience with authoring the pipeline.

Round 1: CI Components maintainer

  • Scenario: You want to release a new version of a CI component and ensure it is properly updated in the CI/CD Catalog.
  • Tasks
    • Creating a project under the private test group
    • Add a simple job for the template
    • Release a new CI component (release-cli)
    • Confirm that it’s on the Catalog
  • Follow-up questions
    • Where would you go to release a new component?
    • What else do you want to include besides the provided test code in your test code?
    • How do you know it's published in the catalog? What information are they looking for and why? if they cannot find it, what's their expectation?

Round 2: CI Components consumer

  • Scenario: You want to integrate a new CI component into your project's pipeline and verify that it meets your requirements.
  • Tasks
    • Create a test project
    • Locate a new component released by the team members in the private test group.
    • Navigate to the test project
    • Add the component to your project’s .gitlab-ci.yml file.
    • Ensure that the component functions as expected and does not introduce any errors in the pipeline.
  • Follow-up questions
    • Where would you go about doing that?
    • How did you validate that you added it to the pipeline?
    • What information is missing?

Schedules

Type Participant Scheduled Access to the group Interviewed
Round 1 Lee ✅ 2024-07-11 ✅ ✅
Round 1 Ahmed ✅ 2024-07-11 ✅ ✅
Round 1 Yasha ✅ 2024-07-15 ✅ ✅
Round 2 Manuel ✅ 2024-07-12 ✅ ✅
Round 2 Ryan ✅ 2024-07-15 Rescheduled to 2024-07-16 ✅ ✅
Round 2 Dominic ✅ 2024-07-15 ✅ ✅
Round 2 Philippe ✅ 2024-07-16 ✅ ✅

Outcomes

  • Having a basic template project to start with would greatly enhance the user experience. (Dovetail insight)
  • There are opportunities to improve the current experience of using input values and authoring the details. (Dovetail insight)
  • Participants want to access version information with history from the catalog. (Dovetail insight)
  • Participants had difficulty locating the CI/CD Catalog menu. (Dovetail insight)
  • Participants questioned why releasing via the UI was not possible. (Dovetail insight)
  • Other topics (Dovetail insight)
Edited by Sunjung Park