#### What did we learn? [Click here for the final report deck](https://www.figma.com/proto/mF555KKsf1m1UyyXbWxXu2/Benchmarking-Slides?node-id=943%3A12915&scaling=scale-down&page-id=40%3A124&starting-point-node-id=943%3A12915) [Video overview](https://youtu.be/cEJaJSuleNA) [](https://www.figma.com/proto/mF555KKsf1m1UyyXbWxXu2/Benchmarking-Slides?node-id=943%3A12915&scaling=scale-down&page-id=40%3A124&starting-point-node-id=943%3A12915) | Pilot Study Results | Link | | ------ | ------ | | Complete data spreadsheet | [Benchmarking pilot spreadsheet](https://docs.google.com/spreadsheets/d/1dNcHC9vxD1EYodFUnII23X8FZPRZ3b_ajyPE_L-B7YY/edit?usp=sharing) | | Dovetail Project | [Dovetail project](https://dovetailapp.com/projects/30oYRVm3wl8esIuImxTdrz/data/b/W7YoDGxZCktQ3hVxMvn8L) | | Recommendation Alignment Mural | [Link](https://app.mural.co/invitation/mural/gitlab2474/1647541370300?sender=u9e0254905830929802683127&key=e77f5557-ec26-42a4-ada9-396f18a67756) | | Other Results | Link | | ------ | ------ | | Creation of a handbook page establishing a usability benchmarking process for GitLab | [Usability benchmarking handbook page](https://about.gitlab.com/handbook/engineering/ux/ux-research-training/usability-benchmarking/) | | Ruby scripts to create test data | [Test data scripting repository](https://gitlab.com/leducmills/uxr-dummy-data-scripts/-/tree/master) | | Creation of a UX cloud sandbox space and handbook page | [UX cloud Sandbox handbook page](https://about.gitlab.com/handbook/engineering/ux/ux-research-training/ux-cloud-sandbox/) | | Video on background / process | [UX Showcase video](https://youtu.be/_u8-EHLy3UI) | #### What’s this epic all about? Today, we don't have a view into how our _present-day_ experiences are performing, from a usability perspective. While we do measure usability via System Usability Scale (SUS), that measure is at the system (aka 'product') level. As a way to better understand and address some of the user feedback we receive (e.g., via SUS or PNPS surveys), we will conduct a usability benchmarking study to identify very specific usability issues and turn them into granular recommendations that can be implemented by product teams to measurably improve usability scores. ### What questions are you trying to answer? GitLab has been measuring the System Usability Scale (SUS) score at the product experience level for many quarters now. While this is an effective approach to measuring the usability of a product, we have had some challenges when trying to identify what to address to ultimately improve the score. Our biggest problem with SUS is that, even though we have themes related to the SUS scores, we don't have a granular view into how our JTBDs are performing within given stages. A usability benchmark process will be established that will focus on the following detailed metrics, for a given task: - Completion rate - Time on task - Customer Effort Score (CES) - Error count - Severity - UMUX Lite - Grade / Overall score Tracking tasks and internally benchmarking them will: - Produce data that reliably and accurately reports the tasks that we select - This data will be able to be analyzed to produce actionable insights - These actionable insights will be used by product teams to inform their roadmap - We will see an improvement in benchmarked tasks over time, based on the actionable insights we produce and implement <details><summary>Results alignment</summary> `2022-03-18` -- The benchmarking analysis has concluded and now it's time to get alignment across the stage on what we learned and what we want to do about it. This is **critically important**, as I do not have enough context and history to know if the recommendations I think are appropriate are already being worked on, or have been discussed and discarded, or are technically not feasible. I have created a mural that explains the background and context for this benchmarking effort and highlights some of the top results and themes. Based on this, I'm asking for input from each of you. Step 1. Please watch **[this video](https://youtu.be/eTSiGAhls2Q)** (6 mins) for some explanation of the results and the exercise. Step 2. Please go to **[this mural link](https://app.mural.co/invitation/mural/gitlab2474/1647541370300?sender=u9e0254905830929802683127&key=e77f5557-ec26-42a4-ada9-396f18a67756)** fill out 'Part 1', and put a checkmark next to your name when you have done so. **Please complete this by `2022-03-30`** If you have any questions, feel free to ask in the issue or contact me on Slack. <table> <thead> <tr> <th>Name</th> <th>Completed</th> </tr> </thead> <tbody> <tr> <td>Marcel van Remmerden @mvanremmerden </td> <td> - [ ] done </td> </tr> <tr> <td> Sarah Waldner @sarahwaldner </td> <td> - [x] done </td> </tr> <tr> <td>Kai Armstrong @phikai </td> <td> - [x] done </td> </tr> <tr> <td>Pedro Moreira da Silva @pedroms </td> <td> - [x] done </td> </tr> <tr> <td>Annabel Dunstone Gray @annabeldunstone </td> <td> - [ ] done </td> </tr> <tr> <td>Amy Qualls @aqualls </td> <td> - [x] done </td> </tr> <tr> <td>Eric Shurter @ericschurter </td> <td> - [x] done </td> </tr> <tr> <td>Michael Le @mle </td> <td> - [x] done </td> </tr> <tr> <td>Mike Nichols @mnichols1 </td> <td> - [x] done </td> </tr> <tr> <td>Torsten Linz @tlinz </td> <td> - [x] done </td> </tr> </tbody> </table> </details> ### Study Profile The benchmarking study was conducted as follows: - 20 participants - External GitLab customers - Variety of experience and roles, focused on [Sasha](https://about.gitlab.com/handbook/product/personas/#sasha-software-developer) persona - Each session will last as long as it needs to (depending on how many tasks we benchmark, this could 90 minutes to 2 hours per session) Tasks were evaluated along the following guidelines: | Measure | Definition | | ------ | ------ | | Completion rate | % of users who successfully complete the task | | Time on task | time spend from start to finish of task | | CES (Customer effort score) | qualitative measure of experienced effort (1-5) | | Severity | Judged priority / importance (per problem / suggestion) | | Overall Score | Given the above measures, a cumulative score will be given for a task | The output, in addition to the Overall Score per task, will be a set of observations and insights, and a granular list of recommendations for product changes / fixes based on the observed evidence that will be delivered to product teams. Example: | Task | Definition | Completion % | Mean Time on Task (avg.) | CES (avg.) | Error Count (avg.) | Score | Severity | Recommendations | | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | | Review code change | Find MR and submit a review | 86% | 2:21 | 3.5 | 2.6 | 76/100 | Medium | Move merge button further from approve button. Minimize comment threads when resolved. | #### Steps These are similar to the [FY23 Q1 KR1 issue](https://gitlab.com/gitlab-org/ux-research/-/issues/1755) around establishing a benchmarking process. - [x] 2022-01-24: @leducmills Conduct 2 pilot usability sessions - [x] 2022-01-31: @leducmills Land on a test environment solution for the pilot study - [x] 2022-01-31: @leducmills Deliver Google Doc draft on testing environment process with .com group - [x] 2022-02-01: @leducmills Begin testing sessions - [x] 2022-02-16: @leducmills Complete testing sessions - [x] 2022-02-17: @leducmills Deliver handbook page on the benchmarking process, detailing out measures - [x] 2022-04-07: @leducmills Deliver handbook page outlining the cloud test environment solution with instructions for people to sign up and use it - [x] 2022-04-07: @leducmills Develop templates / materials for a benchmarking 'kit' - [x] 2022-04-12: @leducmills Conclude pilot study analysis and deliver report to UX and Create stage stakeholders - [x] 2022-04-15: @leducmills Submit iterations for @asmolinski2 and @laurenevans to review - [x] 2022-04-25: @leducmills Update handbook with iterate changes - [x] 2022-04-30: @leducmills Communicate out the usability benchmarking process to UX and Product, including release of benchmarking kit materials ### Tasks to Benchmark The task list can be found in [this Google spreadsheet](https://docs.google.com/spreadsheets/d/1dNcHC9vxD1EYodFUnII23X8FZPRZ3b_ajyPE_L-B7YY/edit?usp=sharing) The study has linked each task to a key JTBD for Create, from one of the three listed below. | Group | Job statement | Source | | ------ | ------ | ------ | | Code Review | When product improvements are identified, I want to propose changes that address them, so that I can help build a better product. | [Issue](https://gitlab.com/gitlab-org/ux-research/-/issues/941) | | Code Review | When my teammates propose changes, I want to review them before they are accepted, so that I can help increase the quality of changes, minimize the risk of defects, minimize the risk of out-of-scope changes, and grow the team’s expertise. | [Issue](https://gitlab.com/gitlab-org/ux-research/-/issues/941) | | Code Review | When my teammates propose changes, I want to ensure they are reviewed and accepted according to internal guidelines, so that we can increase the quality of changes, minimize the risk of defects, and maximize the reliability of product data.| [Issue](https://gitlab.com/gitlab-org/ux-research/-/issues/941) | #### Reviewed literature and background issues. - [Predict task times for primary MR tasks](https://gitlab.com/gitlab-org/ux-research/-/issues/1474) - [MR Performance Epic](https://gitlab.com/groups/gitlab-org/-/epics/6141) - [Collecting MR performance](https://gitlab.com/gitlab-org/ux-research/-/issues/1473) - [Monitoring GitLab.com](https://about.gitlab.com/handbook/engineering/monitoring/) - [Measuring performance](https://about.gitlab.com/handbook/engineering/performance/) - [User-centric performance metrics](https://web.dev/user-centric-performance-metrics/#important-metrics-to-measure) - [ForgePerf - external benchmarking site](https://forgeperf.org/) - [MR usage dashboard](https://app.periscopedata.com/app/gitlab/884002/MR-Related-Usage) - [PNPS Feedback](https://docs.google.com/presentation/d/1Ra_DuI5Uq6aCwDY-pr64qiwT8Oeoz4TWQq9Ag12FgOA/edit#slide=id.g59bfc474c5_2_145) - [Create source code dashboard](https://app.periscopedata.com/app/gitlab/664109/Create:-Source-Code-Dashboard)