UX Scorecard for Runner: Runner Concept Discovery / Understanding
Job To Be Done
JTBD: I need to quickly understand the role of runner in their projects CI/CD, how it works, how to configure it and its underlying features. I should be able to setup builds and other Runner specific setups with ease.
Related JTBD: I need to migrate from existing systems like Jenkins / I need to setup a dedicated runner for my project / I need to inspect an existing job and understand the role runner.
Checklist
Expand/collapse checklist
-
1. Document the current experience of the JTBD, as if you are the user. Capture the screens and jot down observations. Also, apply the following Emotional Grading Scale to document how a user likely feels at each step of the workflow. Add this documentation to the epic's description. -
2. Use the Grading Rubric below to provide an overall measurement that becomes the Benchmark Score for the experience, and add it to the epic's description. -
3. Once you’re clear about the user’s path, create a clickthrough video that walks through the experience and includes narration of the Emotional Grading Scale and Benchmark Score. -
4. Post your video to the GitLab Unfiltered YouTube channel, and link to it from the epic's description. -
5. If your JTBD spans more than one stage group, that’s great! Review your JTBD with a designer from that stage group for accuracy. -
6. Create an issue to revisit the same JTBD the following quarter to see if we have made improvements. We will use the grades to monitor progress toward improving the overall quality of our user experience.
Emotional Grading Scale
Expand/collapse scale
-
🎉 Positive: The user’s experience included a pleasant surprise—something they were not expecting to see. The user enjoyed the experience on the screen and could complete the task, effortlessly moving forward without having to stop and reassess their workflow. Emotion(s): Happy, Motivated, Possibly Surprised -
😐 Neutral: The user’s expectations were met. Each action provided the basic expected response from the UI, so that the user could complete the task and move forward. Emotion(s): Indifferent -
😠 Negative: The user did not receive the results they were expecting. There may be bugs, roadblocks, or confusion about what to click on that prevents the user from completing the task. Maybe they even needed to find an alternative method to achieve their goal. Emotion(s): Angry, Frustrated, Confused, Annoyed
Grading Rubric
Expand/collapse rubric
A (High Quality/Exceeds): Workflow is smooth and painless. Clear path to reach goal. Creates “Wow” moments due to the process being so easy. User would not hesitate to go through the process again.
- Frustration: Minimal to none
- Task Completion: Successful
- Steps to Accomplish Task: Minimal
B (Meets Expectations) Workflow meets expectations but does not exceed user needs. User is able to reach the goal and complete the task. Less likely to abandon.
- Frustration: Low
- Task Completion: Successful
- Steps to Complete Task: Minimal
C (Average) Workflow needs improvement, but user can still finish completing the task. It usually takes longer to complete the task than it should. User may abandon the process or try again later.
- Frustration: Medium
- Task Completion: Successful but with unnecessary steps
- Steps to Complete Task: Average complexity
D (Presentable) Workflow has clear issues and should have not gone into production without more thought and testing. User may or may not be able to complete the task. High risk of abandonment.
- Frustration: High
- Task Completion: Unlikely, but there may be a chance that there is completion
- Steps to Complete Task: Excessive
F (Poor) Workflow leaves user confused and with no direction of where to go next. Can sometimes cause the user to go around in circles or reach a dead end. Very high risk of abandonment, and user will most likely seek other methods to complete the task.
- Frustration: Very High
- Task Completion: Very Unlikely
- Steps to Complete Task: Lacking
Current experience
Runner Concept Discovery Journey
┌──────────────────────────────────┐ │1) User Creates a Project │ └──────────────────────────────────┘ │ │ ▼ ┌──────────────────────────────────┐ ┌──────────────────────────────────┐ │2) GitLab automatically enables │ │3) User follows the banner CTA to │ │Auto-DevOps or prompts the user to│─────▶│setup auto-devops. This takes the │──┐ │set it up on Settings. │ │user to the CI/CD Settings. │ │ └──────────────────────────────────┘ └──────────────────────────────────┘ │ │ │ │ │ ▼ ┌──────────────────────────────────┐ │ ┌─────────────────────────────────┐ │4) User can re-trigger Auto-DevOps│ │ │3) User dismisses Auto-DevOps │ │setup from the project buttons in │ │ │Banner. │ ─────▶│the project home. This takes users│ │ └─────────────────────────────────┘ │to the CI/CD settings. │ │ │ └──────────────────────────────────┘ │ │ │ │ ▼ │ │ ┌─────────────────────────────────┐ │ │ │4) User can follow the "Setup │ │ │ │CI/CD" button in the projects │ │ │ │button. This takes them to to │ │ │ │pre-populated .gitlab-ci.yml │ │ │ │ready to go commit where they can│ │ │ │setup their CI/CD. │ │ │ └─────────────────────────────────┘ │ │ │ │ │ │ │ │ ▼ ▼ │ ┌─────────────────────────────────┐ ┌─────────────────────────────────┐ │ │|| User finds │ │ │ │ │https://docs.gitlab.com/ee/ci/qui│ │|| Runner Settings are available │ │ │ck_start/ where the concept of │ │at the bottom || │ │ │Runners is lightly explained || │ │ │◀─┘ │ │ │(First exposure/touch-point with │ │(First exposure/touch-point with │ │the Runner concept) │ │the Runner concept) │ │ │ ├─────────────────────────────────┴────────┴─────────────────────────────────┤ │ Runner Concept Discovery / First Time User │ └────────────────────────────────────────────────────────────────────────────┘
Runner Concept Discovery Walkthrough (Through Settings)
-
User is prompted to enable AutoDevOps.
Emotional Grade:
🎉 Positive | Why? Discovery of a useful feature with low-setup (first impression)
-
User first exposure to CI/CD Settings / User first exposure to different CI/CD concepts including Runners.
Emotional Grade:
😠 Negative | Why? User is prompted to setup Auto DevOps bit it's launched into a setup experience with limited context or understanding.
-
Runner Settings and Runner Concepts are very vague at first glance.
Emotional Grade:
😠 Negative | Why? Too vague. No initial or clear introduction to the concepts. Hard to understand the role of the Runner
-
Expanding the Runners Settings gives the user a better idea of what’s a Runner, how it's setup and what is doing.
Emotional Grade:
😐 Neutral | Why? Better explanation of the concepts but nothing delightful or outstanding when consuming this information.
-
Use cases for specific runners are not immediately clear to the user from this view.
Emotional Grade:
😠 Negative | Why? This is a potentially confusing area since it's not clear why these settings are necessary or what are they affecting.
-
Same happens for Shared Runners. There’s a semi-clear explanation of how they work and what’s a good fit for them but it doesn’t expand on use cases.
Emotional Grade:
😠 Negative | Why? This is a potentially confusing area since it's not clear why these settings are necessary or what are they affecting.
-
When going to the documentation about Shared Runners, there’s a deeper explanation on how they work.
Emotional Grade:
😐 Neutral | Why? The documentation usually deos a good job of explaining foreign concepts to the user and allows them to create a path to understanding unknown concepts.
-
Back in the settings the list of available runners show some tags. It’s not entirely clear what those tags are indicating.
Emotional Grade:
😠 Negative | Why? No context at all for the tags. The user needs to research more or infer what those tags mean.
-
The ID and number on the right are not very meaningful for a first -time user. Are these references useful, especially in the context of a shared runner which the final user can’t control?
Emotional Grade:
😐 Neutral | Why? Potentially unimportant information at first glance. The user doesn't know if this information is important in the context of the Runner and the underlying jobs.
Runner Concept Discovery Walkthrough (Through Documentation)
-
Users with blank projects will see a CTA to “Set up CI/CD” in their project home.
Emotional Grade:
😐 Neutral
-
After clicking this button the user is prompted to create a .gitlab-ci.yml file. At this point in the journey is possible that the users doesn’t know clearly what this YAML file is for.
Emotional Grade:
😠 Negative | Why? No context at all. User is launched into a blank template for a gitlab-ci.yaml. Users with no context of the role of this file in the CI process are immediately lost.
-
At this point is likely that that that if the user doesn’t know what’s this file they will look it up through Google. The documentation that they find in the first result of Google will give the user its first exposure to the runner concept.
Emotional Grade:
😐 Neutral | Why? Easy to recover from the previous no context situation by looking for the documentation. The documentation is easy to find but doesn't provide any outstanding experience beyond the normal experience you get from technical documentation.
Result
Summary
Discovering the role and use of Runner within the context of CI it's hard. Currently there are no existing paths to configure the Runner. Users who need to setup a dedicated Runner or need to understand the role of a Runner within the context of setting up their CI need to rely heavily on documentation.
Advanced users might already understand the concept of a Runner, especially if they are coming from other CI Server solutions like Jenkins. But this is the type of user who would get more frustrated when not finding a clear path to setup a Specific Runner (dedicated runner).
The existing settings for Runner are very vague and leave the user with many unsolved doubts about how the system works and the differences between the different type of Runners.
The information that is currently surfaced to the user about the Runners in the settings menu is also vague. There are two type of IDs per runner which at first glance are not meaningful or useful for the user. The existing tag system for Runner is also vague and with a high likelihood of being interpreted wrongly by the users.
Grade
D- (Somewhere between a D and an F)
D (Presentable) Workflow has clear issues and should have not gone into production without more thought and testing. User may or may not be able to complete the task. High risk of abandonment.
F (Poor) Workflow leaves user confused and with no direction of where to go next. Can sometimes cause the user to go around in circles or reach a dead end. Very high risk of abandonment, and user will most likely seek other methods to complete the task.
Walkthrough Video
Click Here to See Video At GitLab Unfiltered YouTube Channel