[Feature flag] Rollout of `vulnerability_finding_tracking_signatures`

What

Remove the :vulnerability_finding_tracking_signatures feature flag

Owners

• Team: Secure, Vulnerability Research & Secure, Static Analysis Teams
• Most appropriate slack channel to reach out to: #f_improve_vuln_tracking
• Best individual to reach out to: Ross Fuhrman @rossfuhrman (Backup: Lucas Charles @theoretick)

Expectations

What are we expecting to happen?

We are expecting vulnerability findings to be better tracked as the code of a project changes.

Specifically, changing the line number of a vulnerability finding that is present in the code should no longer cause a new vulnerability to be created/detected.

What might happen if this goes wrong?

Vulnerability management features would be impacted:

• Load/processing times
• Inaccurate vulnerability deduplication

Due to the way !54608 (merged) is implemented, we would be able to fall back to using the current vulnerability tracking methods without losing any data.

What can we monitor to detect problems with this?

https://gitlab.com/gitlab-org/gitlab/-/security/dashboard can be monitored. We should see that a constant rise in the number of vulnerabilities does not occur.

Monitor sidekiq Service | ruby_thread_contention resource Dashboard

Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.

Production:

• gitlab-org/gitlab project
• theoretick/railsgoat
• gitlab-org/security-products/analyzers

Staging:

• rossfuhrman/railsgoat
• theoretick/railsgoat

Roll Out Steps

• Enable on staging (/chatops run feature set feature_name true --staging)
• Test on staging
• Ensure that documentation has been updated
• Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --project=gitlab-org/gitlab feature_name true)

Preparation before production rollout

• Ensure that the feature MRs have been deployed to both production and canary.
  • /chatops run auto_deploy status <merge-commit-of-your-feature>
• Check if the feature flag change needs to be accompanied with a change management issue. Cross link the issue here if it does.
• Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the @sre-oncall Slack alias.
• Ensure that documentation has been updated (More info).
• Announce on the feature issue an estimated time this will be enabled on GitLab.com.
• If the feature might impact the user experience, notify #support_gitlab-com and your team channel (more guidance when this is necessary in the dev docs).
• If the feature flag in code has an actor, enable it on GitLab.com for testing groups/projects.
  • /chatops run feature set --<actor-type>=<actor> <feature-flag-name> true
• Verify that the feature works as expected. Posting the QA result in this issue is preferable.

Global rollout on production

• Incrementally roll out the feature.

  • If the feature flag in code has an actor, perform actor-based rollout.

    • /chatops run feature set vulnerability_finding_tracking_signatures 25 --actors
    • /chatops run feature set vulnerability_finding_tracking_signatures 50 --actors
    • /chatops run feature set vulnerability_finding_tracking_signatures 75 --actors

  • Enable the feature globally on production environment.

    • /chatops run feature set <feature-flag-name> true
• Announce on the feature issue that the feature has been globally enabled.
• Wait for at least one day for the verification term.

Rollback Steps

• This feature can be disabled by running the following Chatops command:

/chatops run feature set --project=gitlab-org/gitlab vulnerability_finding_tracking_signatures false

Open questions

• Do we need a progressive rollout strategy, as in only enable for a percentage of projects per day?
• Can we quantify performance impacts we may see upon rolling out this change (even if it's a guess)?