## Overview Our [research study](https://docs.google.com/presentation/d/1XL0pm_akNinM9eZd0Q1MCTG92bCHHAGCll5LwAVevoU/edit#slide=id.g27cf391daa8_0_0), **reveals a clear customer expectation: machine learning and AI-powered predictions should optimize both CI builds (pipelines and jobs) and the Runner Fleet infrastructure. This represents the strongest signal yet that our product strategy must evolve toward a unified experience integrated into GitLab—one that effectively serves both developer and platform administrator personas.** **Follow-up interviews with enterprise customers confirmed specific needs around CI job performance metrics. Customers require actionable insights into job duration timings to identify optimization opportunities and detect fundamental issues with job definitions.** **Runner Fleet metrics are critical across multiple dimensions. Most importantly, customers need proactive insights into queue wait times, as increases signal either insufficient fleet capacity or underlying infrastructure issues requiring investigation. The key consideration is that customers already maintain extensive observability tooling for their IT systems.** **Therefore, our focus should be providing customers with the essential GitLab CI/CD and Runner Fleet metrics that integrate seamlessly with their existing operational workflows.** Related research <table> <tr> <th>Research issue</th> <th>Insight</th> <th>Dovetail project</th> </tr> <tr> <td> [Audit CI insights existing research](https://gitlab.com/gitlab-org/ux-research/-/issues/2801) </td> <td> **JTBD:** When I am managing continuous integration of code at scale, I want to understand the pipeline health, so I can successfully resolve and prevent issues from occurring. **Problem:** How can I understand the health of my pipeline? Two main buckets: * Status * Speed See full summary of insights in this issue https://gitlab.com/gitlab-org/ux-research/-/issues/2801. </td> <td> :dove: [dovetail insight](https://gitlab.dovetail.com/insights/4TsYg7IFnqdHNaE8gNmCBx) </td> </tr> <tr> <td> https://gitlab.com/gitlab-org/ux-research/-/issues/2617+ </td> <td> **Admins/platform engineers want to see job and pipeline metrics as well as runner metrics in the same view:** When thinking about how to optimize pipelines, runners is a major aspect, but so is the way that the pipeline is configured and even the way the repo is set up. Users need a single place where they can make effective decisions on how to optimize CI/CD, which specifically means including more extensive metrics within the pipeline and job space. </td> <td> [:dove:](https://gitlab.dovetailapp.com/insights/34iGdpt2SQIFKBsOG0w9Er) [dovetail insight](https://gitlab.dovetailapp.com/insights/34iGdpt2SQIFKBsOG0w9Er) </td> </tr> <tr> <td> https://gitlab.com/gitlab-org/ux-research/-/issues/2630+ </td> <td> **Optimization workflows involve trial-and-error and set-and-forget approaches, and common signals of success across Runner and CI Builds. We identified information to reduce the uncertainty users face as they optimize - finding and summarizing this information for users is one way to leverage AI.** The information is across various groups and areas in GitLab and can be found in [this slide](https://docs.google.com/presentation/d/1XL0pm_akNinM9eZd0Q1MCTG92bCHHAGCll5LwAVevoU/edit#slide=id.g27cf391daa8_0_2740). </td> <td> [Slides](https://docs.google.com/presentation/d/1XL0pm_akNinM9eZd0Q1MCTG92bCHHAGCll5LwAVevoU/edit#slide=id.g27cf391daa8_0_0) </td> </tr> </table> ## Brainstorming metrics alignment (2023-11-18) | Category | Question | Metric | Available in GitLab today? | Target persona | Value to persona | Scope | Plans | |----------|----------|--------|----------------------------|----------------|------------------|-------|-------| | CI Insights | What is the total number of pipelines run in my project? | Total pipelines | {project name}\> Analyze\>CI/CD analytics | Sasha - Developer | Low | Project Level | Free, Premium, Ultimate | | CI Insights | What is the total number of successful pipelines in my project? | Count of successful pipelines | {project name}\> Analyze\>CI/CD analytics | Sasha - Developer | Low | Project Level | Free, Premium, Ultimate | | CI Insights | What is the total number of failed pipelines in my project? | Count of failed pipelines | {project name}\> Analyze\>CI/CD analytics | Sasha - Developer | Low | Project Level | Free, Premium, Ultimate | | CI Insights | What is the success rate of the CI/CD pipelines for a project? | Success ratio | {project name}\> Analyze\>CI/CD analytics | Sasha - Developer | Medium | Project Level | Free, Premium, Ultimate | | CI Insights | What are the trends in pipeline duration? | Pipeline duration history | {project name}\> Analyze\>CI/CD analytics | Sasha - Developer | Low | Project Level | Free, Premium, Ultimate | | CI Insights | What’s the 95 percentile CI pipeline duration for a specific time period? | CI pipeline duration 95 percentile | No | Sasha - Develope | Medium | Project Level | Premium, Ultimate | | CI Insights | Which job in the pipeline has the longest duration? | Longest running job per pipeline | No | Sasha - Developer | High | Premium, Ultimate | | | CI Insights | What’s the 95 percentile duration for each job in the project’s pipeline? | CI job duration 95 percentile | No | Sasha - Developer | High | Project Level | Premium, Ultimate | | CI Insights | What’s the 50 percentile duration for each job in the project’s pipeline? | CI job duration 50 percentile | No | Sasha - Developer | High | Project Level | Premium, Ultimate | | CI Insights | What’s the failure rate for a job in the CI pipeline? | CI job failure rate | No | Sasha - Developer | High | Project Level | Premium, Ultimate | | CI Insights | What’s the retry rate for a job in the CI pipeline? | CI job retry rate | No | Sasha - Developer | High | Project Level | Premium, Ultimate | | DORA | | Change failure rate | Value Stream Dashboard | Sasha - Developer | High | Project level | Ultimate | | DORA | | Change lead time | Value Stream Dashboard | Sasha - Developer | High | Project level | Ultimate | | DORA | | Deployment frequency | Value Stream Dashboard | Sasha - Developer | High | Project level | Ultimate | ## Proposal for MVC (revised 2023-12-08): - [ ] Rename CI/CD analytics to CI insights - [ ] Move the DORA metrics (Deployment frequency, Lead time, Time to restore service, change failure rate to Analyze/Value stream analytics - [ ] Focus the new CI insights developer view on critical metrics that provide the following value to developers. - User story: As a developer, I want to pinpoint specific CI jobs that are either failing at a high frequency or the time-to-result duration is exceeding expected targets. ## Wireframe mockups (brainstorm) **Disclaimer** The vision mockups prepared by the design team can, and likely will deviate from the concepts presented below. ### Screen 1: New CI insights view at the project level  ### Screen 2: Drill down into the performance metrics for a specific job