Security release tools: Automated backport merging should check the status of the Default MR
Problem
In recent Security Releases we've picked up unexpected security fixes because the chatops command to merge backports doesn't check the status of the MR targeting the default branch. This means that any fix that becomes ready following release manager checks can be unexpectedly included in the backport merge. Unexpected backports are only caught by chance which puts GitLab.com at risk and adds stress and delay to the security release process.
Intended release steps:
- Release Manager runs checks to validate fixes are ready - only fixes that meet all of the release requirements are considered ready. Requirements include: a full set of MRs, or notification that this fix has reduced backports, approvals, and all MRs assigned to the release bot.
- The Release Manager makes sure all fixes are deployed to Production
- Backports for the same set of fixes are merged using
/chatops run release merge --security
Actual release steps:
This bug causes fixes that become ready between steps 1 and 3 to be picked up by the chatops command. When this happens we merge backports for fixes that haven't been deployed to Production and end up with the following release steps:
- Release Manager runs checks to validate fixes are ready
- The Release Manager makes sure all fixes are deployed to Production
- Backports are merged using
/chatops run release merge --security
- Releasse Manager must notice that unexpected backports have been merged
- Release Manager needs to manually merge the corresponding MRs targeting the default branch and make sure these fixes reach Production before continuing with the release. This adds ~8 hours to the expected Security release preparation
Proposal
The /chatops run release merge --security
command should only merge backports that already have a merged MR targeting the default branch
As a related mitigation step, I'll open an MR to adjust the release issue steps to unlink any excluded fixes from the release before we run the backport merge step.