Automatically sync repos via an MR to complete security releases
Context
One of the final steps of a security release is to sync the Canonical repositories with the security changes. This is done by the /chatops run release sync_remotes --security
command, which syncs the default branch for a number of repositories including: GitLab, GitLab FOSS, Omnibus, Gitaly, etc.
The automatic syncing works for most of the repositories except for GitLab, where it fails with the following:
2022-11-02 14:53:39.005311 F ReleaseTools::RemoteRepository -- Failed to push -- {:remote=>:canonical, :ref=>"master", :output=>"To gitlab.com:gitlab-org/gitlab.git\n ! [remote rejected] master -> master (shallow update not allowed)\nerror: failed to push some refs to 'gitlab.com:gitlab-org/gitlab.git'\n"}
Previous executions of the command:
- https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/8386612
- https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/8111848
The sync failure is currently solved by one of the following:
- Retrying the job until it succeeds - this may not always be possible and creates extra manual work for release managers.
- Merging a generated MR - the release manager will need to find approvers for the MR. The generated MR will also fail current dangerbot rules.
- Force push to master - for compliance reasons, we'll soon change the process around this to create an access request and document evidence. We should consider this a fallback option that is only needed rarely.
Proposal A: Support syncing via a merge request for the GitLab repo
-
Use a gitlab-org
group level bot with maintainer powers, merge rights, and part of the codeowners. -
Ensure the bot from point 1 is part of the different stage team code owner approvals. -
release-tools
creates a merge request using a well-known branch name, as we do for component bumps - gitlab-org/release-tools!2350 (merged). Done via gitlab-org/release-tools!2350 (merged) -
release-tools
will self-approve the MR using the bot from point 1 -
the CI rules for this source branch will be to only run DangerBot
-
dangerbot
will verify that this is a merge from the security mirror authored by our tooling without extra commits on top
If the merge request fails to be created, release managers will create the merge request and ensure the "Squash" option is disabled
Pros
- On an ideal path, release managers won't have to interfere with this process
- Creating a merge request is a compliant operation.
Cons
- Creating a merge request and waiting for it to be merged is a time-consuming process. This adds at least 1.5hrs of waiting
- The merge request is subject to master broken failures.
- Commonly, the merge request could require additional code owner approvals. At this point is up to release managers to find eligible engineers and ask for the MR to be merged.
- If the merge request fails to be created, is up to the release manager to create it (example https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/10244013)
Proposal B: Use merge-train to sync canonical with security changes.
The merge-train could be used to bring security changes into the GitLab canonical repository. Currently, this strategy is used in the opposite direction: to keep the GitLab security repository up to date while the security release is in progress. The same model could be used to sync the GitLab canonical repository after the security release is published:
- Separate the GitLab repository sync from the
sync_remotes
job, the GitLab repository will be handled by the merge train - On the final steps of the security release, release managers will enable the merge train (this could be automated too)
- The merge train will attempt to sync GitLab security code into GitLab canonical repository.
- If it fails, the merge train will try it again in the next
x
minutes. This can be accomplished by checking if the security repo is ahead of canonical, which is the same strategy that is used now to keep the Security repository up to date.
Pros
- Pending on the outcome, release managers don't have to interfere with this process
- No merge request is created, and therefore no need to wait for pipelines or ask for approvals
Cons
- Considering the GitLab canonical traffic, the merge train could take some time to merge the changes.