<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=600301) </details> <!--IssueSummary end--> ## Goal Deliver a lower cost, low effort solution for finding business logic vulnerabilities to our customers that can be used to test demand and willingness to pay. Pending customer feedback, the next iteration could involve a pipeline-based analyzer. ## Beta Success Metrics We will know this beta is a success when customers repeatedly incorporate this Agent into MR review workflows and demonstrably act on its findings, indicating that the Agent delivers trusted, actionable business logic security guidance at acceptable precision and cost. 1. **# of beta customers demonstrating repeat usage after 30 days** - Target \>=10 - Repeat usage can be defined as: customers with developers who @-mention or assign the agent in a MR more than once after their first use. 2. **# of beta customers with findings that received one of the following valued responses before merge within 30 days** (Target \>=10) - Strong signals: - Flagged lines edited - A linked issue created - The MR description updated to note the finding as accepted risk - Weak signal: - Thumbs up 3. **P95 Performance targets:** - @mentions and manual agent assignments to MRs: P95 Target \<=2 minutes ## Beta Exit Criteria 1. Requires at least 10 active customers, of which \>=50% demonstrate repeat month-over-month usage. 2. Requires a named customer quote ## Additional Telemetry 1. Customers with developers using the agent, # of developers using the agent, % of projects with MRs that are using the agent 2. Invocations per developer per week 3. Median and P95 cost per MR scan, broken out by MR size bucket (small/medium/large) 4. Total monthly cost per customer ## Requirements 1. [Per Legal,](https://gitlab.com/gitlab-com/legal-and-compliance/-/work_items/3530) since the agent is an AI-related beta feature, please ensure it is gated behind the admin-level Feature preview toggle. 2. Deliver a reviewer agent in the MR that does a shallow analysis focused on the following vulnerabilities: - Improper Access Control (CWE-284) - Information Disclosure (CWE-200) - Business Logic Errors (CWE-840) - Broken Object-Level Authorization (OWASP API #1, CWE-639) - Broken Function-Level Authorization (OWASP API #3, CWE-862) - Mass Assignment (CWE-915) - Missing Authorization on State-Changing Operations (CWE-862 variant) - Race Conditions in Financial/Stateful Operations (CWE-362/367) 3. Accuracy: given the focus on MR-level analysis, optimize for precision over recall. Precision \>=80% on internal benchmarks. 4. Performance: - @mentions and manual agent assignments to MRs: \<=2 minutes 5. UI changes — prototype: https://slr-agent-prototype-af617c.gitlab.io/ 6. Add necessary metric instrumentation to measure success metrics and additional telemetry above. ## Stretch Goal 1. Provide code suggestions for simple cases