<!-- The first section "Release notes" is required if you want to have your release post blog MR auto generated. Currently in BETA, details on the **release post item generator** can be found in the handbook: https://about.gitlab.com/handbook/marketing/blog/release-posts/#release-post-item-generator and this video: https://www.youtube.com/watch?v=rfn9ebgTwKg. The next four sections: "Problem to solve", "Intended users", "User experience goal", and "Proposal", are strongly recommended in your first draft, while the rest of the sections can be filled out during the problem validation or breakdown phase. However, keep in mind that providing complete and relevant information early helps our product team validate the problem and start working on a solution. --> # :clock: Updates We meet once a week, typically on Mondays, in the **Security AI Weekly Sync**. For updates please review the [latest agenda](https://docs.google.com/document/d/1jpW0us_THrzsh5eJ1PuSevNx2miaLlRbSHg_E0owOeI/edit#heading=h.jzva6sn1lfs4) and/or recording. Beta is currently on track to ship in %"16.3". ### Release notes <!-- What is the problem and solution you're proposing? This content sets the overall vision for the feature and serves as the release notes that will populate in various places, including the [release post blog](https://about.gitlab.com/releases/categories/releases/) and [Gitlab project releases](https://gitlab.com/gitlab-org/gitlab/-/releases). " --> GitLab surfaces vulnerabilities that contain relevant information, however, sometimes it is unclear where to start. It takes time to research and synthesize information that is surfaced within the vulnerability record. Moreover it can be difficult to figure out how to fix a given vulnerability. With this release you can click a button to get an explanation and recommendation on how to mitigate the vulnerability. ### Problem to solve <!-- What problem do we solve? Try to define the who/what/why of the opportunity as a user story. For example, "As a (who), I want (what), so I can (why/value)." --> In https://gitlab.com/groups/gitlab-org/-/epics/10284+ we created a way for users to quickly understand a vulnerability so that they know what next steps to take, i.e. what code change do I need to make etc. We have received feedback from the market and seen competition announce similar features; GitHub has [CopilotX for the Entire Dev Workflow](https://www.youtube.com/watch?v=ZtZ0xdk5wTM) and Snyk has [AI-generated security fixes](https://snyk.io/blog/ai-generated-security-fixes-in-snyk-code-now-available/). Explain this Vulnerability will mature into Beta. ### Focus for Beta - Get feedback from the user as to how effective the AI response is in their workflow. - Increase the security and performance of this feature by exposing the prompt, doing a pre-flight check to check for secrets, and include additional monitoring, alerting and code coverage. - Create a tool to test prompt/responses for different models so that we can make an informed decision on which model and prompt to use for GA. **Why aren't we focusing on the prompt/response for Beta?** - Some models that we are considering using for this feature aren't available yet. - We want to get feedback from users. - There have been some issues with content moderation and outages with the current model that we are using. We want to make sure we can quickly troubleshoot this feature sooner rather than later. ### Focus to GA - Refine the prompt and response to be sure users are getting a useful response that helps them in their workflow. ### Intended users - [Sasha (Software Developer)](https://about.gitlab.com/handbook/product/personas/#sasha-software-developer) can use this feature to better understand and potentially fix vulnerability findings before she tries to merge to the default branch. - [Sam (Security Analyst)](https://about.gitlab.com/handbook/product/personas/#sam-security-analyst) uses this feature to quickly triage vulnerabilities and learn about specific vulnerabilities quickly. ### Requirements <!-- What is the single user experience workflow this problem addresses? For example, "The user should be able to use the UI/API/.gitlab-ci.yml with GitLab to <perform a specific task>" https://about.gitlab.com/handbook/product/ux/ux-research-training/user-story-mapping/ --> - [x] For SAST vulnerabilities, a user can click a button on a vulnerability that auto-generates a LLM prompt: - [x] Explains a vulnerability - [x] Recommends what needs to be changed in the code - [x] Give input if the suggestion was helpful or not with a :thumbsup_tone1: or :thumbsdown_tone1:, https://gitlab.com/gitlab-org/gitlab/-/issues/412753+s ~"closed::complete" - [x] Provide input as to how the answers could have been better. https://gitlab.com/gitlab-org/gitlab/-/issues/413693+s ~"workflow::blocked" - [x] Responses will be given in a timely manner and cached if necessary - [x] https://gitlab.com/gitlab-org/gitlab/-/issues/409143+s - [x] https://gitlab.com/gitlab-org/gitlab/-/issues/412713+s - [x] Ability to see the prompt for debugging purposes - [x] https://gitlab.com/gitlab-org/gitlab/-/issues/407757+s - [x] The feature is secure and is not at risk for things like prompt injections - [x] https://gitlab.com/gitlab-org/gitlab/-/issues/412712+s - [x] Behind a feature flag that is on by default and the UI reflects Beta status. - [ ] Issue TBC - [x] Behind a toggle that is off by default and the UI reflects Beta status - [x] https://gitlab.com/gitlab-org/gitlab/-/issues/412960+s - [x] There is no longer going to be separate experiment/beta toggles, it will all be behind a single pre-GA toggle: https://gitlab.com/groups/gitlab-org/-/epics/10537#new-requirements ## Further details <!-- Include use cases, benefits, goals, or any other details that will help us understand the problem better. --> ### Documentation requirements for Beta [Documentation for Beta features](https://docs.gitlab.com/ee/policy/alpha-beta-support.html#beta) lists several requirements that will need to be verified including: - [x] Configuration and dependencies unlikely to change. - [x] Features and functions unlikely to change. However, breaking changes may occur outside of major releases or with less notice than for Generally Available features. - [x] Documentation reflects Beta status. - [x] UX complete or near completion. - [x] Behind a feature flag that is on by default and the UI reflects Beta status, _included in the [requirements section](https://gitlab.com/groups/gitlab-org/-/epics/10368#requirements)_ - [x] Behind a toggle that is off by default and the UI reflects Beta status, _included in the [requirements section](https://gitlab.com/groups/gitlab-org/-/epics/10368#requirements)_ ## Product requirements for Beta We also need to adhere to [internal criteria for maturing an Experiment to a Beta feature](https://internal-handbook.gitlab.io/handbook/product/ai-strategy/ai-integration-effort/prioritization/#criteria-for-maturing-experimental-features-to-beta): - [x] You have received explicit approval from your stage leader and Hillary Benson or David DeSanto to mature your feature to Beta. - [x] Requirements for Beta are met. _See check list section immediately above this section._ - [x] [AI model guidance](https://internal-handbook.gitlab.io/handbook/product/ai-strategy/ai-integration-effort/ai_model_guidance/) is followed. _We are proceeding with Google PaLM `codechat-bison-001` which has been approved for Beta features._ - [x] Feature is gated by a pre-GA namespace toggle and the [third-party services toggle](https://docs.gitlab.com/ee/user/group/manage.html#group-third-party-ai-features-setting), _included in the [requirements section](https://gitlab.com/groups/gitlab-org/-/epics/10368#requirements)_ - [x] Feature’s documentation explicitly names the third-party AI service provider and model powering the feature (e.g. Google PaLM text-bison-001). - [x] The Product manager DRI has confidence that this feature solves a meaningful customer problem. - [ ] UI for the feature has been approved by Legal. - [ ] [Requirements from Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/feature-support.html) for supporting this level of feature are met ## UX requirements for Beta - [x] [Problem Validation](https://internal-handbook.gitlab.io/handbook/product/ai-strategy/ai-integration-effort/ux_maturity/#validation-problem-validation) is documented and includes a mix of evidence and assumptions. https://gitlab.com/gitlab-org/gitlab/-/issues/414323+ - [x] [Solution Validation](https://internal-handbook.gitlab.io/handbook/product/ai-strategy/ai-integration-effort/ux_maturity/#validation-solution-validation) is given a Grade C or higher and is evaluated in half a day by a Product Designer. https://gitlab.com/gitlab-org/gitlab/-/issues/414324+ - [x] [Success metrics](https://internal-handbook.gitlab.io/handbook/product/ai-strategy/ai-integration-effort/ux_maturity/#build-improve) are defined by the team. Please see the [Feature Usage Metrics](https://gitlab.com/groups/gitlab-org/-/epics/10368#feature-usage-metrics) section in this epic. - [ ] [Design standards](https://internal-handbook.gitlab.io/handbook/product/ai-strategy/ai-integration-effort/ux_maturity/#design-standards) are adhered to ### Permissions and Security <!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)? Consider adding checkboxes and expectations of users with certain levels of membership https://docs.gitlab.com/ee/user/permissions.html * [ ] Add expected impact to members with no access (0) * [ ] Add expected impact to Guest (10) members * [ ] Add expected impact to Reporter (20) members * [ ] Add expected impact to Developer (30) members * [ ] Add expected impact to Maintainer (40) members * [ ] Add expected impact to Owner (50) members Please consider performing a threat model for the code changes that are introduced as part of this feature. To get started, refer to our Threat Modeling handbook page https://about.gitlab.com/handbook/security/threat_modeling/#threat-modeling. Don't hesitate to reach out to the Application Security Team (`@gitlab-com/gl-security/appsec`) to discuss any security concerns. --> ### Documentation <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/workflow.html#for-a-product-change * Add all known Documentation Requirements in this section. See https://docs.gitlab.com/ee/development/documentation/workflow.html * If this feature requires changing permissions, update the permissions document. See https://docs.gitlab.com/ee/user/permissions.html --> - Add specific documentation that links to https://gitlab.com/gitlab-org/gitlab/-/issues/407294+ ### Availability & Testing <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the Quality Engineering quad planning and test planning processes and reach out to your counterpart Software Engineer in Test for assistance. Quad Planning: https://about.gitlab.com/handbook/engineering/quality/quality-engineering/quad-planning Test Planning: https://about.gitlab.com/handbook/engineering/quality/quality-engineering/test-engineering/#test-planning --> ### Available Tier <!-- This section should be used for setting the appropriate tier that this feature will belong to. Pricing can be found here: https://about.gitlab.com/pricing/ * Free * Premium/Silver * Ultimate/Gold --> ~"GitLab Ultimate" ### Feature Usage Metrics <!-- How are you going to track usage of this feature? Think about user behavior and their interaction with the product. What indicates someone is getting value from it? Create tracking issue using the Snowplow event tracking template. See https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Snowplow%20event%20tracking.md --> - [x] Track the event count for Explain this Vulnerability in the [AI Features Dashboard](https://app.periscopedata.com/app/gitlab/1137231/Ai-Features). This includes a daily count and a total count. - [x] Understand the Explain Vulnerabilities Funnel in the [AI Feature Use Dashboard](https://app.periscopedata.com/app/gitlab:safe-dashboard/1141091/Ai-Experiments-Dashboard). The funnel includes: - Number of Users that View Vulnerability Report - Number of Users that View a Vulnerability record - Number of Users that View explain_vulnerability - Number of explain_vulnerability multi day users - [x] Get user feedback from https://gitlab.com/gitlab-org/gitlab/-/issues/412753+ and https://gitlab.com/gitlab-org/gitlab/-/issues/413693+. ### What does success look like, and how can we measure that? <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. Create tracking issue using the Snowplow event tracking template. See https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Snowplow%20event%20tracking.md --> * The event count for Explain this vulnerability increases over time, users are engaging this this feature. * The number of suggestions that are helpful increases over time, making the number of questions answered that weren't helpful nominal. ### Is this a cross-stage feature? <!-- Communicate if this change will affect multiple Stage Groups or product areas. We recommend always start with the assumption that a feature request will have an impact into another Group. Loop in the most relevant PM and Product Designer from that Group to provide strategic support to help align the Group's broader plan and vision, as well as to avoid UX and technical debt. https://about.gitlab.com/handbook/product/#cross-stage-features --> No, not at this time. ### What is the competitive advantage or differentiation for this feature? ### Links / references <!-- Label reminders - you should have one of each of the following labels. Use the following resources to find the appropriate labels: - Use only one tier label choosing the lowest tier this is intended for - https://gitlab.com/gitlab-org/gitlab/-/labels - https://about.gitlab.com/handbook/product/categories/features/ --> ```mermaid --- title: Explain this Vulnerability --- sequenceDiagram autoNumber actor User participant Banner as UI Banner participant Drawer as UI Drawer participant Abstraction as Abstraction Layer participant API as AI API participant API2 as AI API 2 User->>Banner: Click Try Button Banner->>Drawer: Load Drawer Drawer--)+Abstraction: Send Vulnerability ID and include_code Abstraction->>Abstraction: Get Vulnerability Data Abstraction->>Abstraction: Get File Contents Abstraction->>Abstraction: Construct Prompt alt Secondary AI Vendor Enabled Abstraction--)+API2: Send Prompt API2->>-Abstraction: Return response else Primary AI Vendor Abstraction--)+API: Send Prompt API->>-Abstraction: Return response end Abstraction->>-Drawer: Render AI response ``` <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->