Generate auto-deploy branches every six hours
In #1104 (closed) we discussed using the merge train to automatically update auto-deploy branches more frequently. In this discussion it was brought up that we should maybe just generate auto-deploy branches more frequently.
This brings me to my proposal: we should adjust Release Tools so that we create an auto-deploy branch every six hours, instead of once per day.
The benefits here are more frequent automated staging and canary deploys, as well as removing the need for Release Managers to remember updating auto-deploy branches if they want to deploy more frequently. For this to work, we do need to implement #1120 (closed) so that we don't end up with even more branches than we have today; especially if we want to create branches even more frequently than every six hours.
I propose we do so at the following hours:
| Deploy | Americas (UTC-9) | Americas (UTC-6) | Time UTC | EU time (UTC+2) |
|---|---|---|---|---|
| 1 | 19:00 | 22:00 | 04:00 | 06:00 |
| 2 | 01:00 | 04:00 | 10:00 | 12:00 |
| 3 | 07:00 | 10:00 | 16:00 | 18:00 |
| 4 | 13:00 | 16:00 | 22:00 | 00:00 |
Bold cells indicate the deployment takes place during reasonable (ish) working hours.
These timings are not arbitrary: it takes about 2 hours to produce a package, and an additional 2 hours or so to get this on canary. This means that around 10:00 UTC we'd have the first canary deploy. With production deploys taking around 2 hours as well, this means the earliest production deploy is done around 12:00 UTC. Similarly, the second deploy would be done around 18:00.
For Release Managers in the Americas this would be similar, with the first deploy being done around 12:00, and the second one being done around 18:00. Of course this does vary based on where in the Americas you are based. But even in UTC-9 the deploy times would be reasonable.
This allows for at least two deployments per day for Release Managers, without having to wake up crazy early or stay until late in the evening.
TODO
-
Adjust Release Tools to include the hour in branch names: gitlab-org/release-tools!1129 (merged) -
Enable the feature flag auto_deploy_hours -
Manually run the prepare pipeline: https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/1607857 -
Ensure parts of the build process (e.g. Omnibus) all work -
Adjust the Release Tools schedule to run every six hours
Outcome
Since May 2020 Average MTTP has been close to our target of 24hrs. To assist Release Managers in making multiple deployments each day the frequency of the branch creation was increased. (update 2020-09-10 - timeframes for branch creation have been adjusted down to every 8 hours to better suit release management shifts)