Review late 13.9 requests - How can we improve release stability?
In the run up to 13.9 we had several late requests to pick changes into the release. Late changes mean additional work for release managers and can risk the stability of the release.
Use this issue to understand the timeline and identify improvements we can make for future releases.
Timeline of events, times in UTC:
- 2021-02-16 - 13:17 - @nolith announces the commit guaranteed for release on 22nd (Slack link - internal)
- 2021-02-16 - 14:59 @marin notifies release managers that gitlab-org/gitlab!54317 (merged) needs to be picked into the release.
- 2021-02-17 - 09:32 - @aakriti.gupta notifies about an issue that could potentially break upgrades gitlab-org/gitlab#321734 (closed)
- 2021-02-17 - 14:21 - Fix for breaking upgrades picked into the auto-deploy gitlab-org/gitlab!54439 (comment 511214309)
- 2021-02-17 - 21:58 - @mayra-cabrera announces the candidate commit to be released on 22nd (Slack link - internal)
- 2021-02-18 - 15:30 - @10io notifies release managers of a breaking MR that is not in the release candidate but will need a fix MR included if the release candidate changes.
- 2021-02-18 - 21:12 - @mayra-cabrera announces final commit for release on 22nd (Slack link - internal)
- 2021-02-19 - 16:11 - @fabiopitino notifies release managers of an S2 regression that will require a patch - gitlab-org/gitlab!54669 (merged)
- 2021-02-19 - 18:09 - @andr3 notifies release managers of an S2 bug that will require a patch - gitlab-org/gitlab!54671 (merged)
- 2021-02-19 - 23:02 - @lkerr notifies release managers of an S2 regression that will require a patch - gitlab-org/gitlab!54736 (merged)
@aakriti.gupta @10io @fabiopitino @andr3 @lkerr - thanks for bringing these issues to the attention of release managers. In this release we were lucky with timings and severities and were able to complete the release preparation steps on time. But, in the interest of improving release stability could you each leave a comment to help us understand what we need to do to reduce the risk from late-occurring bugs?
@grantyoung @pks-t gitlab-org/gitlab!54317 (merged) was the highest risk issue we found in the run up to this release and also one that comes from a slightly different process. Was there a way we could have caught this issue earlier, or taken a safer approach in the run-up to the release? Ideally, as the release candidate had already been announced this issue would have been flagged up in the #releases channel to make sure release managers aware, is there anything we can do to improve this process?