Experiment: Merging security merge requests targeting 'master' between July 31st - August 4th
On the Time-to-merge statistics for security merge requests, we observed that 83% of the security MRs prepared between December 2019 to July 2020 were ready to go ~6 days before the respective Security Release. This places the accumulation point of such MRs one week before the Security Release starts.
This means if we merge security MR's when ready, the divergence between Canonical master
and Security master
is only going to be essentially present during the last week of the month and practically non-existing for the rest of the month.
Proposal
We can use this uneven spread to our advantage and experiment by merging security merge requests that are considered ready by our tooling between August 3rd and August 4th, two days before the Security Release.
Merging process will be like:
- Iterate over the security issues associated with the Security Release Tracking Issue - https://gitlab.com/gitlab-org/gitlab/-/issues/225575
- For those security merge requests targeting
master
that are ready, we proceed to merge them into Securitymaster
- Merged security merge requests are going to be picked into the current Security auto-deploy branch.
- Auto-deploy branch is deployed to Staging, Canary and Production as usual.
Notes
- Merging security fixes on
Security
, implies thatmaster
and auto-deploy branches between Canonical and Security are going to diverge during this period of time. - Auto-deploy branches should be created from Security
master
for the duration of the experiment.
Steps
✅
Completed Decisions
Topic | Link |
---|---|
Merging security fixes against security master
|
#1018 (comment 379321208) |
Merge requests that should be merged | #1018 (comment 379320758) |
Decide update strategy between Canonical and Security diverged branches | #1040 (closed) |
Deployment Tracking should continue tracking merge requests on Canonical | #1031 (closed) |
Run the experiment | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1074 |
Implementation Details
Topic | Link |
---|---|
Auto-deploy branches need to be created on Canonical and/or Security | #943 (closed) && #1026 (closed) (Output may depend on the Deployment Tracking resolution) |
Configure merge train to update Security master
|
#1073 (closed) |
Ensure only GitLab Security issues are processed | https://gitlab.com/gitlab-org/release-tools/-/issues/472 |
Modify our batch merging tooling process to consider merged security MR's | https://gitlab.com/gitlab-org/release-tools/-/issues/471 |
Implement merging security MR's targeting master once they're ready (backports should not be merged |
#958 (closed) |
Implement picking merged security MR's into the current Security auto-deploy branch | #958 (closed) |
⚪
Blockers
⏳
In progress
⏭
Next Implementation details
Unsolved questions
- What would happen if a file is modified at the same time on Canonical and on Security (with different changes on each repo), and merge train tries to merge the changes?
- Given we only plan to run the experiment for a couple of days, this may be an edge case.
- Fixing this will require human intervention, so we should prepare a runbook stating how do we recover from the breakage
Highlights
- We started on Friday rather than Monday which gave us one more day and the entire weekend to observe.
- It didn't reduce reduced the auto-deploy pace
- On Friday, we did 3 deployments to prod
- On Monday, we did 2 deployments to prod
- On Tuesday, we did 3 deployments to prod
- Before the security release started,
10
security merge requests targeting master were processed, and9
of those were deployed to production. - The Security Release included 14 security issues, only
4
were processed during the actual release:- 1 belonging to another project (not GitLab Security)
-
3
were not processed given they had less than4
MR associated and our tooling ignore those (this is independent of the experiment).
- Merging and deploying MR's earlier cause for the steps on the security template to be considerably reduced https://gitlab.com/gitlab-org/release/tasks/-/issues/1509
- Merging Canonical master into Security master through merge train was seamless and it didn't cause any broken master issue.
Conclusion
This experiment allowed multiple GitLab.com deployments during the time we would normally be frozen to prepare the monthly Security release. Keeping MTTP well below our target as well as allowing security fixes going out to GitLab.com as soon as they’re ready.
I believe we can use the same strategy in a larger window of time during the release cycle, meaning
- Auto-deploy branches will live in Security from now on.
- Allow for security merge requests targeting
master
to be merged during a larger window of time. - Automatically pick security fixes into the auto-deploy current branch.
- Deploy the security fixes at the same pace as regular fixes
- Update Security
master
branch with the security merge-train.
Next Steps
- Update &109 (closed) with the above proposal
Investigation
Topic | Link |
---|---|
Investigate automatic toggling of security merge train based on mirror status | #1111 (closed) |
Allow for security merge requests to trigger Pipelines for merged results | #1107 (closed) |
Implementation details
Topic | Link |
---|---|
Create auto-deploy branches solely on Security mirrors | #1101 (closed) |
Modify the way release-tools processes security merge requests | #1122 (closed) |
Change the way security implementation issues are closed | #1106 (closed) |
Add documentation about how to deal with conflicts caused by security merge train | #1123 (closed) |