Explain this Vulnerability: Prompt / Response export tool for analysis
Why are we doing this work
This issue formally tracks the development of a tool that exports prompt/responses for continued analysis. The thread at #412538 (comment 1409485379) provides context.
The tool should use https://gitlab.com/gitlab-org/security-products/tests/webgoat.net as it's base project. This project contains 165 intentional SAST (via Semgrep) vulnerabilities
Sheet location
Sheet columns:
Automated
Export a CSV for the vulnerabilities in this project: https://gitlab.com/gitlab-org/govern/threat-insights-demos/personal-test-projects/webgoat.net
Capture
- Id
- Severity
- Identifiers
- Prompt
- Language
- code_bison
- chat_bison
- result of
"safetyAttributes"=>{"blocked"=>**true**}
to aid in Responsible AI moderation refinement - API response time
- result of
- text_bison
- anthropic
- openai_completion
- openai_chat
Results will be imported into the sheet at https://docs.google.com/spreadsheets/d/1KMN3_8g6SIwG9J-pS7S7kIzixdOnYVpoMp2qTLJy1Bw/edit#gid=689565258
Relevant links
Non-functional requirements
-
Documentation: -
Feature flag: -
Performance: -
Testing:
Implementation plan
Verification steps
Edited by Neil McCorrison