Create a process for addressing bugs introduced by security merge requests
Context
Reverting a security merge request is not encouraged because it rollbacks a security fix compromising the integrity of GitLab.com and self-managed instances. However, it has happened that a security merge request introduces a bug, and a such requires a mitigation strategy. Examples:
- https://gitlab.com/gitlab-org/gitlab/-/issues/393774#note_1331187212
- https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8606#note_1335427849
- https://gitlab.com/gitlab-org/gitlab/-/issues/423920#note_1579465573
- https://gitlab.com/gitlab-org/gitlab/-/issues/393774#note_1331849377
- https://gitlab.com/gitlab-org/gitlab/-/issues/416654#note_1491630843
The mitigation options will depend on the severity of the security issue, the impact of the bug introduced, and the timeline of the security release.
Options
Option 1 - The bug introduced is fixed:
This is the ideal option but depends on the proximity of the security release due date, if the due date is in less than 48hrs, this option is not available.
The stage team prepares a bug fix, merges it, and ensures it is deployed to GitLab.com at least 48hrs before the security release due date. This option guarantees the security issue and the bug fix are included in the security release.
Note The security release due date can't be delayed for compliance reasons and to avoid impacting the monthly release schedule.
Option 2 - Fully in - Security merge request is not reverted
Suggested option for high-severity security issues: severity1, severity2
Pending on the impact of the bug, the security merge request and the backports are included in the security release, guaranteeing the vulnerability is fixed on GitLab.com and on self-managed installations.
Once the security release is out:
- Stage team fixes the bug in the [GitLab canonical] repo.
- Stage team backports the bug fix to the current stable version following the [patch release process].
Option 3 - Fully out - Security merge request is reverted and removed from the security release
Suggested options for low-severity security issues: severity3 and below
The security merge request is reverted and removed from the security release. This implies the security vulnerability will not be fixed on GitLab.com and self-managed installations, and the details about the vulnerability will be leaked once the security release is out. This option requires explicit approval from AppSec
Proposal - Introduce a process for addressing bug introduced by security merge requests
-
Document the possible options for addressing a bug fix introduced by a security fix -
Add a merge request template on the GitLab repository to be used when the revert option is chosen (option 3) - gitlab-org/gitlab!116615 (diffs) -
The template should be updated to include a checkbox for AppSec stating they approve the vulnerability to be disclosed without a fix.
-
-
Link the new runbook to https://about.gitlab.com/handbook/engineering/releases/#how-can-i-revert-a-security-merge-request -
Add the new runbook to the Delivery changelog https://gitlab.com/gitlab-org/release/docs/-/blob/master/CHANGELOG