<!-- The first section "Release notes" is required if you want to have your release post blog MR auto generated. Currently in BETA, details on the **release post item generator** can be found in the handbook: https://about.gitlab.com/handbook/marketing/blog/release-posts/#release-post-item-generator and this video: https://www.youtube.com/watch?v=rfn9ebgTwKg. The next four sections: "Problem to solve", "Intended users", "User experience goal", and "Proposal", are strongly recommended in your first draft, while the rest of the sections can be filled out during the problem validation or breakdown phase. However, keep in mind that providing complete and relevant information early helps our product team validate the problem and start working on a solution. --> ### Release notes <!-- What is the problem and solution you're proposing? This content sets the overall vision for the feature and serves as the release notes that will populate in various places, including the [release post blog](https://about.gitlab.com/releases/categories/releases/) and [Gitlab project releases](https://gitlab.com/gitlab-org/gitlab/-/releases). " --> ### Problem to solve <!-- What problem do we solve? Try to define the who/what/why of the opportunity as a user story. For example, "As a (who), I want (what), so I can (why/value)." --> I've learned to maintain my own CI/CD pipelines, writing code, deploying the app, and throughout the process, installing dependencies needed inside CI/CD jobs. Sometimes, there is a failure within the job logs. I need to read and parse the error log, often a stacktrace from a script or command that does not run properly. Or it is a single logged error line that requires research and analysis. In a recent example, the deployment of my opsindev.news newsletter to GitLab Pages failed, because of a regression in the MkDocs Material Theme being installed with pip. https://gitlab.com/dnsmichi/opsindev.news/-/jobs/3544293987  The Python module is installed from a private repository, using an insiders subscription that enables social plugins and GDPR cookie banners. https://gitlab.com/dnsmichi/opsindev.news/-/blob/main/.gitlab-ci.yml <details><summary>Click to expand</summary> ```yaml # base installs the requirements .mkdocs-base: script: - pip install -r requirements.txt # build generates static site, throws errors with wrong config, etc. .mkdocs-build: script: - mkdocs build --site-dir public # insiders uses the MkDdocs Material insiders version (sponsoring by dnsmichi, token in CI/CD variables) .mkdocs-insiders: script: - !reference [.mkdocs-base, script] - | if [ -z "${GITHUB_TOKEN_MKDOCS}" ]; then echo "Using MkDocs Material default" else echo "Using MkDocs Material insiders" pip install git+https://${GITHUB_TOKEN_MKDOCS}@github.com/squidfunk/mkdocs-material-insiders.git fi - !reference [.mkdocs-build, script] # lint and generate site on main branch pages: extends: .mkdocs-insiders stage: deploy only: - main artifacts: paths: - public ``` </details> The procedure to fix the pipeline is 1. Copy the error message into a web search, `AttributeError: 'SocialPlugin' object has no attribute '_image_promises'` 2. Additionally, open the project's bug tracker and search for `SocialPlugin` 3. Found the problem in https://github.com/squidfunk/mkdocs-material/issues/4819 4. Maintainer commented with a fix, linking to a commit/PR, labeled `fix available`. 5. Solution is to wait for a new nightly build of the MkDocs material pypi package and later re-run the CI/CD deployment. This needs an issue to track its due date, and increase visibility (if there were more team members).  To analyse the problem, I need to leave the GitLab UI and job log and open several new tabs. The strategy on searching for an error in CI/CD can be overwhelming - from long experience, and as a developer, it is easier to identify the words to copy into a web search, a beginner may not know what a stacktrace is. Suppose the maintainer would not have pushed a fix already. In that case, I'd need to create an upstream issue myself, and also an issue/incident in GitLab that the CI/CD pipelines are failing at the moment - and the fix is pending, waiting for upstream to release a fixed version, and tracking the issue URL in the incident. Another use case are errors on www-gitlab-com with failed blog posts deployments (YAML frontmatter errors) or other linting errors that may require team members to ask for help. ### Intended users Different personas, from experienced CI/CD pipeline builders to users who run into blog deployment troubles. <!-- Who will use this feature? If known, include any of the following: types of users (e.g. Developer), personas, or specific company roles (e.g. Release Manager). It's okay to write "Unknown" and fill this field in later. Personas are described at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/ --> * [Delaney (Development Team Lead)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead) * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) * [Priyanka (Platform Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#priyanka-platform-engineer) * [Sidney (Systems Administrator)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator) * [Rachel (Release Manager)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#rachel-release-manager) * [Alex (Security Operations Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#alex-security-operations-engineer) * [Simone (Software Engineer in Test)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#simone-software-engineer-in-test) * [Allison (Application Ops)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#allison-application-ops) * [Ingrid (Infrastructure Operator)](https://about.gitlab.com/handbook/product/personas/#ingrid-infrastructure-operator) * [Dana (Data Analyst)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#dana-data-analyst) * [Eddie (Content Editor)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#eddie-content-editor) ### User experience goal <!-- What is the single user experience workflow this problem addresses? For example, "The user should be able to use the UI/API/.gitlab-ci.yml with GitLab to <perform a specific task>" https://about.gitlab.com/handbook/product/ux/ux-research-training/user-story-mapping/ --> The user should be able to fix a failing CI/CD pipeline and job fast. Adding documentation on how to fix the problem should be easy from the UI, with additional AI help that knows how to troubleshoot more efficient or already provide the possible solution. ### Proposal <!-- How are we going to solve the problem? Try to include the user journey! https://about.gitlab.com/handbook/journeys/#user-journey --> This is a proposal with multiple ideas: 1. Being able to comment on a job log line (similar to code reviews), and document how to fix the problem. - The error may come again, which can help create a knowledge base if this becomes a well established team workflow. - Maybe this comment can be copied/persisted somewhere else (a documentation issue), because job logs can get huge/slow/deleted - Or create an alert/incident from the job UI, so that the analysis and learning process is transparent for everyone. 2. GitLab CI/CD learned from past pipeline runs that these errors frequently come again, and offers a help blurb in the job log UI on how to fix the problem. - Thinking of MLOps/AIOps here, similar to the detection of flaky tests, and more Observability data available. - For example, if the error is related to network timeouts, or generic HTTP errors, additional metrics from CI/CD environments could help increase pipeline efficiency. - If a code stacktrace is rendered, the AI can detect the language, used package, bug tracker, and offer immediate research help (and maybe a code change to fix). The first iteration can be better rendering of the error message, i.e. marking it bold. ### Further details <!-- Include use cases, benefits, goals, or any other details that will help us understand the problem better. --> "Fix failed pipelines fast" is a common requirement for efficient pipelines. Avoiding errors, and detecting them early enough, with fast help to fix them, can help developers, DevOps engineers, Ops, etc. to become more efficient. In the future world, AI may help solve these problems. This requires more pipeline insights (OpenTelemetry data, and additional instrumentation of job script steps), and training history of the user's projects to learn specific problem types. - https://gitlab.com/gitlab-org/gitlab/-/issues/338943+ - https://gitlab.com/gitlab-org/gitlab/-/issues/373143+ - https://gitlab.com/groups/gitlab-org/opstrace/-/epics/41+ Providing the option to document, learn and "auto fix" pipelines also enables beginners and first time contributors, who are not familiar with the code base, nor the architecture and its dependencies that may cause errors in complex CI/CD pipelines. Last but not least, this could also become a way to better troubleshoot CI/CD pipelines, and create a collection of known CI/CD problems (needs research on data processing, user agreements, etc.). Other possibilities are better documentation for pipeline efficiency, and learning complex pipelines. - https://gitlab.com/gitlab-org/gitlab/-/issues/367062+ - https://gitlab.com/gitlab-org/verify-stage/-/issues/360+ ### Permissions and Security The ability to comment on job logs, or get AI tips should be limited to logged in users, with project access. <!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)? Consider adding checkboxes and expectations of users with certain levels of membership https://docs.gitlab.com/ee/user/permissions.html * [ ] Add expected impact to members with no access (0) * [ ] Add expected impact to Guest (10) members * [ ] Add expected impact to Reporter (20) members * [ ] Add expected impact to Developer (30) members * [ ] Add expected impact to Maintainer (40) members * [ ] Add expected impact to Owner (50) members Please consider performing a threat model for the code changes that are introduced as part of this feature. To get started, refer to our Threat Modeling handbook page https://about.gitlab.com/handbook/security/threat_modeling/#threat-modeling. Don't hesitate to reach out to the Application Security Team (`@gitlab-com/gl-security/appsec`) to discuss any security concerns. --> ### Documentation 1. How to create an issue/incident from commenting on a job log (depends how this is implemented) 2. How to navigate AI assisted tips in CI/CD - use case examples. <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/workflow.html#for-a-product-change * Add all known Documentation Requirements in this section. See https://docs.gitlab.com/ee/development/documentation/workflow.html * If this feature requires changing permissions, update the permissions document. See https://docs.gitlab.com/ee/user/permissions.html --> ### Availability & Testing 1. Evaluate whether commenting on job logs, and persisting the data "somewhere" affects performance. Linking issues with job log lines can affect the frontend rendering for example. (context: I remember that we added a regex to render URLs in job logs that caused troubles). 2. Define how much AI assisted help is possible, and clarify it in the documentation. <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the Quality Engineering quad planning and test planning processes and reach out to your counterpart Software Engineer in Test for assistance. Quad Planning: https://about.gitlab.com/handbook/engineering/quality/quality-engineering/quad-planning Test Planning: https://about.gitlab.com/handbook/engineering/quality/quality-engineering/test-engineering/#test-planning --> ### Available Tier <!-- This section should be used for setting the appropriate tier that this feature will belong to. Pricing can be found here: https://about.gitlab.com/pricing/ * Free * Premium/Silver * Ultimate/Gold --> * Free: Ability to comment on job logs and create issues/alerts. * Ultimate: Advanced AI assisted tips for more efficient pipelines, "auto fix", etc. * Should also be available to contributors that fork an Ultimate project, and create an upstream MR that runs the pipelines. ### Feature Usage Metrics <!-- How are you going to track usage of this feature? Think about user behavior and their interaction with the product. What indicates someone is getting value from it? Create tracking issue using the Snowplow event tracking template. See https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Snowplow%20event%20tracking.md --> Failed jobs and pipelines, and actions taken after viewing the job log. Is a new issue/alert created? Was the AI assist help blurb used to create a follow-up MR to fix the pipeline? ### What does success look like, and how can we measure that? Month-over-month reduction of failing pipelines, and patterns of failure being reduced. Dogfooding on https://gitlab.com/gitlab-com/www-gitlab-com with the handbook, website and blog deployments provides a great measurement for failing pipelines. Common failures are linter errors (handbook links, codeowners, redirects, blog front matter yaml). Examples in these issues: - https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/13991+ - https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/13980+ - https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10326+ https://gitlab.com/gitlab-org/gitlab provides complex pipelines and potential failure reduction too. Another measurement could be: - First time contributors that fork the project and create a MR - number of CI/CD pipeline failures, and how they are reduced with additional help. This could also help increase the number of contributors, when waiting for a pipeline, and fixing things is easier and faster. <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. Create tracking issue using the Snowplow event tracking template. See https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Snowplow%20event%20tracking.md --> ### What is the type of buyer? <!-- What is the buyer persona for this feature? See https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/buyer-persona/ In which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#three-tiers --> ### Is this a cross-stage feature? <!-- Communicate if this change will affect multiple Stage Groups or product areas. We recommend always start with the assumption that a feature request will have an impact into another Group. Loop in the most relevant PM and Product Designer from that Group to provide strategic support to help align the Group's broader plan and vision, as well as to avoid UX and technical debt. https://about.gitlab.com/handbook/product/#cross-stage-features --> Yes. Needs discussion on the feature scope and iterations. There might be overlaps with AI Assist, Observability and Pipeline Efficiency initiatives already planned. I think that the verify stage in the ops section can be the DRI, cc @jreporter - ~"group::pipeline execution" - ~"section::ops" @jheimbuck_gl - ~"group::runner" - ~"section::ops" @DarrenEastman - ~"group::pipeline insights" - ~"section::ops" @jocelynjane - ~"group::observability" - ~"section::ops" @kbychu - ~"group::mlops" - ~"section::data-science" @tmccaslin Quality engineering and contributor success teams may also be interested. cc @meks @nick_vh ### What is the competitive advantage or differentiation for this feature? Pipeline Efficiency on one platform: Actions on jobs logs (document, alert) and AI assisted tips to fix pipelines fast. Pipeline Efficiency is a complex topic, with different possibilities to analyse, research, learn and solve. After giving talks around this topic in 2020 and 2021, I recognized a growing interest in 2022 at KubeCon EU and NA in [lightning talks](https://docs.google.com/presentation/d/1k4PWKJR9O1jEGxKblSQtjDsloQ95uvu6Ty9Pjpmin7E/edit) at the GitLab booth. I will continue researching and creating proposals and content in FY24 (2023). ### Links / references - https://gitlab.com/gitlab-org/gitlab/-/issues/338943+ - https://gitlab.com/gitlab-org/gitlab/-/issues/373143+ - https://gitlab.com/groups/gitlab-org/opstrace/-/epics/41+ - https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/13991+ - https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/13980+ - https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10326+ - KubeCon NA Lightning Talk "Efficient DevSecOps Pipelines in a Cloud Native World" at GitLab booth by @dnsmichi: https://docs.google.com/presentation/d/1k4PWKJR9O1jEGxKblSQtjDsloQ95uvu6Ty9Pjpmin7E/edit <!-- Label reminders - you should have one of each of the following labels. Use the following resources to find the appropriate labels: - https://gitlab.com/gitlab-org/gitlab/-/labels - https://about.gitlab.com/handbook/product/categories/features/ --> <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->