🤖 [Experiment] GitLab Duo Vulnerability Triage

Experiment summary

In %16.1 we completed https://gitlab.com/gitlab-org/gitlab/-/issues/409134+ but found that the responses were not useful. (For more details, please see the conclusion. We believe that the combination of updated models, increased token limits/context windows and the addition of data like KEV, EPSS, CVSS will enable us to provide users with a better response.

To verify that, we will enhance the prompt that we used for the original experiment with these additional parameters. And review the responses to see if they have improved and are useful.

Hypothesis

Vulnerability triage is a hard problem to solve programmatically. Generative AI can help give a user an idea of where to start triaging their vulnerabilities.

Business problem

GitLab's vulnerability report surfaces all vulnerabilities on the default branch. More often users aren't sure where to start. What vulnerability puts my organization most at risk? Which vulnerability should I work on first?

Supporting data

Expected outcome

Experiment design & implementation

ICE score

Impact Confidence Ease Score
value 1 value 2 value 3 Average(1:3)

Known assumptions

Results, lessons learned, next steps

Checklist

  • Fill in the experiment summary and write more about the details of the experiment in the rest of the issue description. Some of these may be filled in through time (the "Result, learnings, next steps" section for example) but at least the experiment summary should be filled in right from the start.
  • Add the label of the group:: that will work on this experiment (if known).
  • Mention the Product Manager, Engineering Manager, and at least one Product Designer from the group that owns the part of the product that the experiment will affect.
  • Fill in the values in the ICE score table ping other team members for the values you aren’t confident about (i.e. engineering should almost always fill out the ease section). Add the ~"ICE Score Needed" label to indicate that the score is incomplete.
  • Replace the ~"ICE Score Needed" with an ICE low/medium/high score label once all values in the ICE table have been added.
  • Mention the [at]gitlab-core-team team and ask for their feedback.
Edited by Alana Bellucci