17.5 Duo Enterprise Testing Checklist

This checklist can be used to test the AI Vulnerability Explanation and Resolution for the Duo Enterprise testing requirements.

• Spreadsheet of test cases

Reading Materials

• Vulnerability Explanation (VE): https://docs.gitlab.com/ee/user/application_security/vulnerabilities/#vulnerability-explanation
• Vulnerability Resolution (VR): https://docs.gitlab.com/ee/user/application_security/vulnerabilities/#vulnerability-resolution

License:

• Both VE and VR belongs to the "Duo Enterprise" license.

Testing Steps

Feature Flags

All the necessary FF have already been enabled by default, so you shouldn't have to worry about this.

• vulnerability_resolution_ai_gateway
• vulnerabiliyt_resolution_ga
• resolve_vulnerability_ai

Setting Configuration

1. Create a new group
2. Clone projects:
   • Has high confidence CWE: https://staging.gitlab.com/govern-team-test/oxeye-rulez
   • (optional), has a mix of vulnerabilities: https://gitlab.com/gitlab-examples/security/security-reports
3. Group > Settings > General > Permissions and group features > Check "Use GitLab Duo features"

Vulnerability Explanation Expectation

1. Click on a SAST vulnerability
2. Click on "Explain with AI"
3. It triggers the duo chat drawer calling the /vulnerability_explain command
4. It successfully display the AI response
5. ✅ Success

Vulnerability Resolution Expectation

1. Click on a SAST vulnerability

For high confidence CWE:

2. It displays the button "Resolve with AI"
3. When clicked, it creates the AI generated MR
4. ✅ Success

GA_VR

For all other SAST:

2. It displays the button "Resolve with AI"
3. It is in a disabled state and has a "Learn more" link
4. ✅ Success

For high confidence CWE	For all other SAST
	-

Notes

• AI features are for namespaces only, since individual users cannot be on an ultimate plan and/or buy add-ons. Therefore, you must clone the project within a group or namespace.
• Since 17.3, both VE and VR are not longer "experiments" and therefore not tied to the "GitLab Duo experiment and beta features" setting.

Resources

• Previous tests:
  • Instruction: https://gitlab.com/gitlab-org/fulfillment/meta/-/work_items/1942#note_2022375962
  • Result for Duo Enterprise: https://gitlab.com/gitlab-org/fulfillment/meta/-/issues/1871#note_2043143487
• Smoke tests for Beta VR (functionally GA):
  • 17.3: https://gitlab.com/gitlab-org/gitlab/-/issues/474232+