Keep track of deployment advancement and production status
This issue is about extending release-tools
to keep track of the ongoing deployment.
With #1094 (closed) we will trigger production checks at specific checkpoints during a production deployment. (TL;DR: implementation recap in #1241 (comment 434063792))
We need a proper way to handle those checks in release-tools.
At the moment, the baking time message is the closest thing to what we need
Proposed implementation
-
Modify
ReleaseTools::Tasks::AutoDeploy::CheckProduction
to accept some additionalENV
parameters-
DEPLOYMENT_CHECK
: will be set to"true"
to trigger the deployment advanced status -
DEPLOYMENT_STEP
: a string describing in which point of the deployment we are at the moment (this is url-encoded) -
DEPLOYER_JOB_URL
: the URL of the deployer job triggering the check
-
-
If
DEPLOYMENT_CHECK==true
a newReleaseTools::Promotion::Manager#deployment_check
method will be invoked -
The baking time foreword is replaced with a specific message
:an_emoji_that_makes_sense: Production deployment reached ${DEPLOYMENT_STEP} Package version: `the version` Pipeline: ${DEPLOYER_JOB_URL}
-
Rework the summary line with something like
The deployment can continue/Please @release-manager investigate if it's worth halting the deployment
-
checks are ok in the current format
To do
-
Wrap up implementation gitlab-org/release-tools!1254 (merged) -
Enabled the feature flag and notify teamDelivery -
deployment_check
FF forrelease-tools
- https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/185/edit -
Add TRIGGER_INTERMEDIATE_PRODUCTION_CHECKS
ENV variable to thedeployer
project - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/settings/ci_cd
-
-
Fix DEPLOYER_JOB_URL
andDEPLOYMENT_STEP
variables -
Complete #1346 (closed) -
Only send notifications if the deployment is in progress and the production checks are failing -
Move the notification to the #announcement
Slack channel
-
-
Limit the notification to be sent only in deployments to prod - gitlab-org/release-tools!1304 (closed) / https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/merge_requests/324 -
Document TRIGGER_INTERMEDIATE_PRODUCTION_CHECKS
env variable https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/variables.md#deployer - gitlab-org/release/docs!304 (merged) -
Wrap the notifications in a single Slack thread - #1372 (closed) -
Open follow ups: - To remove the feature flag - #1373 (closed)
- To restore the release managers mention #1443
Development log
- Initial implementation was done on gitlab-org/release-tools!1254 (merged)
- Messages were considered noisy by the team because of it:
- Messages are only sent if a deployment is on-going and if there's an active incident (or a problem with health status) #1346 (closed)
- Messages should be wrapped up in Slack threads #1372 (closed)
- Testing was successfully completed on #1432 (closed), you can see some examples on #1432 (comment 472549089)
- Notifications were moved to
f_upcoming_release
channel on gitlab-org/release-tools!1334 (merged)
How to turn it off?
- Disable the feature flag on release tools https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/185/edit
- Remove
TRIGGER_INTERMEDIATE_PRODUCTION_CHECKS
from deployer - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/settings/ci_cd