UX Scorecard - Verify:Pipeline Authoring FY21Q3 - Pipeline authoring experience

JTBD

I want to author a CI pipeline so others in my team can leverage CI to increase the efficiency of their tasks.

User story: I want to create a GitLab CI pipeline to deploy my new website.

Evaluation type: Self-heuristic evaluation.

• Personas: Sasha (Software Developer), Devon (DevOps Engineer)
• Previous score and scorecard: NA
• Benchmark score: Grade D - Poor
• Walkthrough video: https://www.youtube.com/watch?v=xx-Bxm4qA_I&feature=youtu.be
• Recommendations: #1391 (closed)

UX Scorecard Checklist

Learn more about UX Scorecards

Expand/collapse checklist

1. Add this issue to the stage group epic for the corresponding quarter's UX scorecards. Verify that the "UX scorecard" label is applied.
2. After working with your PM to identify a top job, write it using the Job to Be Done (JTBD) format: When [situation], I want to [motivation], so I can [expected outcome]. Review with your manager to ensure your JTBD is written at the appropriate level. Remember, a JTBD is not a user story and should not lead directly to a solution.
3. Make note of which personas might be performing the job, and link to them from this issue's description. Keeping personas in mind allows us to make the best decisions to address specific problems and pain points. Note: Do not include a persona in your JTBD format, as multiple types of users may complete the same job.
4. If your JTBD spans more than one stage group, that’s great! Review your JTBD with a designer from that stage group for accuracy.
5. Review the current experience, noting where you expect a user's high and low points to be. Capture the screens and jot down observations.
   • If you're re-scoring the experience, recapture the entire flow. You will likely have some of the artifacts (i.e. a UI screen that wasn't changed) that you can simply reuse.
6. Have internal or external users accomplish the JTBD. Record this session and document their experience. Note that 3-5 users are preferred, as this provides valuable insights and removes subjectivity. Make sure to avoid setting up a task-based usability study. The goal is to provide the participant context (the JTBD) and listen and watch how they attempt to complete the job. What we learn may differ from participant to participant.
   • If it's not possible to have internal or external users go through the experience, a self-heurisitic evaluation can be done instead.
7. Use the Grading Rubric to provide an overall measurement that becomes the Benchmark score for the experience (one grade per JTBD), and add it to this issue's description. Document the score in the UX Scorecard Spreadsheet.
8. Once testing is complete, create a walkthrough video that documents what you experienced/witnessed within the existing experience. Begin the video with a contextual introduction including: your role, stage group, specifiy how you acquired the data (ex: internal or external users, or self-heurisitic evaluation), and a short introduction to your JTBD and purpose of the UX scorecard. This is not a "how to" video, but instead should help build empathy for users by clearly showing areas of potential frustration and confusion. (You can point out where the experience is positive, too.) At the end of the video, make sure to include narration of the Benchmark Score. Examples here and here.
   • If you're re-scoring the experience, walkthrough the entire flow again. For narration, you can highlight the recent improvements but still call out any areas that could still use some tweaking (in the next round of iterations, if applicable). The re-score video, in theory, should be shorter since we've hopefully eliminated a few bumps in the user flow.
9. Post your video to the GitLab Unfiltered YouTube channel, and link to it from this issue's description.
10. Link to your video in the Engineering Week in Review.
11. Create a recommendation issue for this JTBD and add it to the same stage group epic as this issue. Also add a link to your recommendation issue to this issue.
12. Following the UX Scorecards setup instructions, create an issue (and epic, if needed) to rescore the same JTBD the following quarter to see if we have made improvements. We will use the grades to monitor progress toward improving the overall quality of our user experience. Add that issue as related to this issue.

Emotional Grading Scale

Expand/collapse scale

• 🎉 Positive: The user’s experience included a pleasant surprise—something they were not expecting to see. The user enjoyed the experience on the screen and could complete the task, effortlessly moving forward without having to stop and reassess their workflow. Emotion(s): Happy, Motivated, Possibly Surprised
• 😐 Neutral: The user’s expectations were met. Each action provided the basic expected response from the UI, so that the user could complete the task and move forward. Emotion(s): Indifferent
• 😠 Negative: The user did not receive the results they were expecting. There may be bugs, roadblocks, or confusion about what to click on that prevents the user from completing the task. Maybe they even needed to find an alternative method to achieve their goal. Emotion(s): Angry, Frustrated, Confused, Annoyed

Experience Walkthrough

User story: I want to create a GitLab CI pipeline to deploy my new website.

Expand/collapse the walkthrough with screenshots

1. 😐 Neutral: Where do I start? Not sure how is Auto DevOps different from CI? Well, I don't know anything about CI, but I know I want to try it, so let's try "Set up CI/CD".

2. 😠 Negative: Hmm, no idea what's happening. I wanted to set up CI/CD, but I was taken to this empty file. Is this a bug? It doesn't say anything about CI. Oh wait, it's a .gitlab-ci.yml file. Guess it's where the configuration should be, so I need to write it by hand? How? Is there documentation anywhere?

3. 😠 Negative: Oh, there are some templates. I barely noticed them. But how do I know which template is right for me? Also, the templates don't make much sense without knowing the syntax. Guess I should Google some help.

4. Aha, there's a Getting Started link! Hope it helps me!

5. 😐 Neutral: Okay, so I need to create that CI file and set up a runner. Let's see what I need to do about that runner.

6. 😠 Negative: This is very confusing and I have no idea if the runner is set up. It says the shared runners are available, but do they need to be set up? Guess let's leave things as they are and I probably will be notified if something about the runner isn't set up.

7. 😐 Neutral: I have a website, so let's google how I can deploy it with GitLab CI. Not sure what's a Pages website, but let's see since it's the first result.

8. 😐 Neutral: Aha, so there's an HTML template there, I haven't noticed it at first. This seems like the best first step I can think of.

9. 😐 Neutral: This is a very basic template, probably I need to customize it. But I don't know anything about how to define this file, so let's just try as is just to see what happens.

10. 😠 Negative: So where do I see the pipeline? Is it working?... Let's check the navigation.

11. 😐 Neutral: Aha, here is the pipeline page, and the pipeline is green, guess it's working. Let's try clicking on it.

12. 😐 Neutral: Cool, I can see the pipeline now! How do I see the logs and all the details of what happened?

13. 😐 Neutral: Found the logs! Everything looks like it's working, but I'm not sure. How do I access the website though? Looks like it's been deployed but there's no link anywhere...

14. 😐 Neutral: Let's see. Job artifacts, maybe there's something there?

15. (not included in grading) 😐 Neutral: Oh here's the new index file!

16. (not included in grading) 🎉 Positive: Wow, everything actually worked so smoothly in the background but I had no idea about it! Took me ages to figure out this thing, but it's working really well!

Results

Summary

From the POV of a new user who's not familiar with CI, the experience was very difficult and disjointed. The UI has no guidance on the next steps at all, the only way to move forward is to read the documentation from multiple pages.

The templates are difficult to discover, and the only way to see what that template does it to apply it. The templates could include more guidance/ tips/ help in the yaml comments as well.

After committing the yaml configuration, there's no feedback as to what's happening with the pipeline, and no clear path to navigate to the pipeline page to see its status.

The feature for deploying a website using pages through an HTML template is quite incredible. It was perfectly executed, but there was zero confirmation in the UI. There was no way to know where the website deployed to.

We have a rich library of templates that work well, but we're not leveraging their full potential because the value is not clearly communicated and it's unclear how to use them and how to understand when the configuration worked or not.

Because of poor feedback, it's difficult to troubleshoot the configuration. Navigating between the editor and the pipeline page is not easy either.

Grade

☝ The last few steps were not included in calculation of the grade since they're not directly related to the pipeline authoring experience.

Grade D - Poor

Experience is viewed as a poor experience and is difficult to complete.

• Ease: Difficult
• Experience: Bad