Rollback tracking jobs in deployer should be non-blocking
The deployer triggers a release-tools pipeline that records a Deployment in various projects on gitlab.com. It's currently blocking in nature, so if gitlab.com is down or very slow as a result of an incident, these jobs have the potential to fail a deployment pipeline, including rollbacks.
The bulk of the work for the tracking job is in release-tools on Ops so moving to non-blocking should be enough to de-risk. However, this job currently does more than just tracking deployments and is also generating the QA issue which is then required to trigger QA pipelines. Until we've split this functionality out of the tracking job let's make this job non-blocking on rollbacks only. This will reduce the risk of .com being down and us being unable to rollback due to this job.
Once we've split QA issue generation off this job we can change to simply confirm that release-tools created the pipeline, and then move on with the deployment. It doesn't need to wait for a successful pipeline.
Proposal
These are the operations performed by the *-track-running and *-track jobs:
| Item | Blocking in regular pipeline | Required in rollback pipeline | Blocking in rollback pipeline | Notes | |
|---|---|---|---|---|---|
| 1 | Records/updates a deployment in GitLab, Omnibus, Gitaly security projects. | No | Yes | No | |
| 2 | Adds workflow:: label to MRs. |
No | No | - | |
| 3 | Creates QA issue. | Yes | No | - | The QA issue seems to be required for QA reports based on #1490 (comment 742851134). We can consider making it non-blocking as well. |
| 4 | Creates a Sentry deploy. | No | Yes | No | |
| 5 | Records/updates a deployment in https://ops.gitlab.net/gitlab-org/release/metadata. | Yes | Yes | Yes |
Items 3 and 5 are to be moved into separate rake tasks that can be called by independent blocking jobs. After that, we can make the tracking jobs non-blocking.
-
Create separate rake tasks in release-tools to record a deployment in release/metadata, and to create a QA issue - gitlab-org/release-tools!1680 (merged). -
Create separate jobs in deployerthat call the above rake tasks - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/458. -
Make the trackjobs (indeployer) non-blocking for regular and rollback pipelines - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/462.
Cleanup:
-
Cleanup the track_deploymentrake task by removing the creation of the metadata deployment and QA issue - gitlab-org/release-tools!1688 (merged). -
Remove the no_metadata_trackingandno_qavariables fromdeployer- https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/468.
Follow-up:
- Make the
create_qa_issuejob non-blocking as well? We might be better off tackling this in a separate issue.create-qa-issuedoesn't run in rollback pipelines though, so we might not need to worry about this.