Introduce release manager and deploy manager rotation
This issue is a proposal to discuss a less stressful working style for release managers.
Current schedule
Currently, release managers are responsible for on-premise packages and deployment to GitLab.com
The following diagram represents (abusing the gantt color schema) a release manager shift. Tasks in light gray are outside of a monthly shift.
gantt
title Release Management - on-premise packages
dateFormat MM-DD
todayMarker off
axisFormat %b, %d
section Previous milestone 14.10
preparation :done, old_monthly, 04-15 , 7d
14.10 :done, milestone, m1, 04-22,
14.10.2 :p1, after sec1, 1d
14.10.X patch releases window :after sec1, 24d
section Current milestone 15.0
preparation :monthly, 05-15, 7d
15.0 : milestone, m2, 05-22,
section Security release
Preparation :prep_old, after old_monthly, 4d
14.10.1 :sec1, after prep_old, 1d
14.9.X :after prep_old, 1d
14.8.X :after prep_old, 1d
Preparation :done, prep, after monthly, 4d
15.0.1 :done, sec2, after prep, 1d
14.10.3 :done, after prep, 1d
14.9.X+1 :done, after prep, 1d
Critical security release window : after sec1, 24d
While the above diagram covers a monthly shift, auto_deploy packages follow a faster pace, the following diagram it's an approximation of a single day (diagram generated with $2325011)
This is a big graph, please scroll at the end and press the "Display" button to show the picture
gantt
title Release Management - auto-deploy
dateFormat HH:mm
axisFormat %H:%M
todayMarker off
Section Auto-Deploy 00 UTC
New Auto-Deploy branch: milestone, m00, 00:00,2min
tag :tag00, 00:20, 5min
omnibus package :omnibus00, after tag00, 1h
helm charts : charts00, after tag00, 20m
gstg-cny :deploy_gstg-cny00, after omnibus00 charts00, 1h
gstg-ref :deploy_gstg-ref00, after omnibus00 charts00, 1h
QA :gstg-cny-qa00, after deploy_gstg-cny00, 1h
gprd-cny :deploy_gprd-cny00, after gstg-cny-qa00, 1h
baking time :baking00, after deploy_gprd-cny00, 1h
gstg :deploy_gstg00, after baking00, 2h
delay gprd :delay_gprd00, after baking00, 30m
gprd :deploy_gprd00, after delay_gprd00, 2h
Section Auto-Deploy 03 UTC
New Auto-Deploy branch: milestone, m03, 03:00,2min
tag :tag03, 03:20, 5min
omnibus package :omnibus03, after tag03, 1h
helm charts : charts03, after tag03, 20m
gstg-cny :deploy_gstg-cny03, after omnibus03 charts03, 1h
gstg-ref :deploy_gstg-ref03, after omnibus03 charts03, 1h
QA :gstg-cny-qa03, after deploy_gstg-cny03, 1h
gprd-cny :deploy_gprd-cny03, after gstg-cny-qa03, 1h
baking time :baking03, after deploy_gprd-cny03, 1h
gstg :deploy_gstg03, after baking03, 2h
delay gprd :delay_gprd03, after baking03, 30m
gprd :deploy_gprd03, after delay_gprd03, 2h
Section Auto-Deploy 06 UTC
New Auto-Deploy branch: milestone, m06, 06:00,2min
tag :tag06, 06:20, 5min
omnibus package :omnibus06, after tag06, 1h
helm charts : charts06, after tag06, 20m
gstg-cny :deploy_gstg-cny06, after omnibus06 charts06, 1h
gstg-ref :deploy_gstg-ref06, after omnibus06 charts06, 1h
QA :gstg-cny-qa06, after deploy_gstg-cny06, 1h
gprd-cny :deploy_gprd-cny06, after gstg-cny-qa06, 1h
baking time :baking06, after deploy_gprd-cny06, 1h
gstg :deploy_gstg06, after baking06, 2h
delay gprd :delay_gprd06, after baking06, 30m
gprd :deploy_gprd06, after delay_gprd06, 2h
Section Auto-Deploy 09 UTC
New Auto-Deploy branch: milestone, m09, 09:00,2min
tag :tag09, 09:20, 5min
omnibus package :omnibus09, after tag09, 1h
helm charts : charts09, after tag09, 20m
gstg-cny :deploy_gstg-cny09, after omnibus09 charts09, 1h
gstg-ref :deploy_gstg-ref09, after omnibus09 charts09, 1h
QA :gstg-cny-qa09, after deploy_gstg-cny09, 1h
gprd-cny :deploy_gprd-cny09, after gstg-cny-qa09, 1h
baking time :baking09, after deploy_gprd-cny09, 1h
gstg :deploy_gstg09, after baking09, 2h
delay gprd :delay_gprd09, after baking09, 30m
gprd :deploy_gprd09, after delay_gprd09, 2h
Section Auto-Deploy 11 UTC
New Auto-Deploy branch: milestone, m11, 11:00,2min
tag :tag11, 11:20, 5min
omnibus package :omnibus11, after tag11, 1h
helm charts : charts11, after tag11, 20m
gstg-cny :deploy_gstg-cny11, after omnibus11 charts11, 1h
gstg-ref :deploy_gstg-ref11, after omnibus11 charts11, 1h
QA :gstg-cny-qa11, after deploy_gstg-cny11, 1h
gprd-cny :deploy_gprd-cny11, after gstg-cny-qa11, 1h
baking time :baking11, after deploy_gprd-cny11, 1h
gstg :deploy_gstg11, after baking11, 2h
delay gprd :delay_gprd11, after baking11, 30m
gprd :deploy_gprd11, after delay_gprd11, 2h
Section Auto-Deploy 14 UTC
New Auto-Deploy branch: milestone, m14, 14:00,2min
tag :tag14, 14:20, 5min
omnibus package :omnibus14, after tag14, 1h
helm charts : charts14, after tag14, 20m
gstg-cny :deploy_gstg-cny14, after omnibus14 charts14, 1h
gstg-ref :deploy_gstg-ref14, after omnibus14 charts14, 1h
QA :gstg-cny-qa14, after deploy_gstg-cny14, 1h
gprd-cny :deploy_gprd-cny14, after gstg-cny-qa14, 1h
baking time :baking14, after deploy_gprd-cny14, 1h
gstg :deploy_gstg14, after baking14, 2h
delay gprd :delay_gprd14, after baking14, 30m
gprd :deploy_gprd14, after delay_gprd14, 2h
Section Auto-Deploy 17 UTC
New Auto-Deploy branch: milestone, m17, 17:00,2min
tag :tag17, 17:20, 5min
omnibus package :omnibus17, after tag17, 1h
helm charts : charts17, after tag17, 20m
gstg-cny :deploy_gstg-cny17, after omnibus17 charts17, 1h
gstg-ref :deploy_gstg-ref17, after omnibus17 charts17, 1h
QA :gstg-cny-qa17, after deploy_gstg-cny17, 1h
gprd-cny :deploy_gprd-cny17, after gstg-cny-qa17, 1h
baking time :baking17, after deploy_gprd-cny17, 1h
gstg :deploy_gstg17, after baking17, 2h
delay gprd :delay_gprd17, after baking17, 30m
gprd :deploy_gprd17, after delay_gprd17, 2h
Section Auto-Deploy 20 UTC
New Auto-Deploy branch: milestone, m20, 20:00,2min
tag :tag20, 20:20, 5min
omnibus package :omnibus20, after tag20, 1h
helm charts : charts20, after tag20, 20m
gstg-cny :deploy_gstg-cny20, after omnibus20 charts20, 1h
gstg-ref :deploy_gstg-ref20, after omnibus20 charts20, 1h
QA :gstg-cny-qa20, after deploy_gstg-cny20, 1h
gprd-cny :deploy_gprd-cny20, after gstg-cny-qa20, 1h
baking time :baking20, after deploy_gprd-cny20, 1h
gstg :deploy_gstg20, after baking20, 2h
delay gprd :delay_gprd20, after baking20, 30m
gprd :deploy_gprd20, after delay_gprd20, 2h
The daily activities can easily be overwhelming. The above diagram is just a reference, but we have to take into consideration incidents, QA failures, pipeline glitches, and people asking for information about the release or the deployment.
New proposal
I propose we split the release manager rotation into release managers and deploy managers.
The first one will be a one-month-long shift, taking care of releasing packages, and the latter will be a two-week-long shift taking care only of GitLab.com auto-deploy packages.
deploy manager - shift duration: 2 weeks (maybe 1 is enough)
A deploy manager takes care of every auto_deploy package. They receive a handover message at the beginning of their day, and if nothing special happened, they will take care of promoting the packages.
A deploy manager will take care of the following things:
- incidents related to a deployment
- coordination to deploy fixes in a timely manner
- rollbacks
- coordination to handle QA smoke test failures
release manager - shift duration: 1 month
A release manager takes care of every public package release. They are making the plan for the monthly release, the security release, and any other release needed during the month.
A release manager will take care of the following things:
- planning the monthly release week
- planning the security release
- monitoring the status of picks and planning for patch releases
- coordination with quality to make sure stable branches are green to handle backports and patches
- coordination with the release-post manager
Point of contact between the two roles
During a security release and during the preparation for the monthly release, the two roles require extra coordination, we may make the assumption that during those two weeks the release managers will also serve as deploy managers. Outside of those weeks, a release manager cannot be in the rotation as a deploy manager.