Rollback tracking jobs in deployer should be non-blocking

The deployer triggers a release-tools pipeline that records a Deployment in various projects on gitlab.com. It's currently blocking in nature, so if gitlab.com is down or very slow as a result of an incident, these jobs have the potential to fail a deployment pipeline, including rollbacks.

The bulk of the work for the tracking job is in release-tools on Ops so moving to non-blocking should be enough to de-risk. However, this job currently does more than just tracking deployments and is also generating the QA issue which is then required to trigger QA pipelines. Until we've split this functionality out of the tracking job let's make this job non-blocking on rollbacks only. This will reduce the risk of .com being down and us being unable to rollback due to this job.

Once we've split QA issue generation off this job we can change to simply confirm that release-tools created the pipeline, and then move on with the deployment. It doesn't need to wait for a successful pipeline.

Proposal

These are the operations performed by the *-track-running and *-track jobs:

Item Blocking in regular pipeline Required in rollback pipeline Blocking in rollback pipeline Notes
1 Records/updates a deployment in GitLab, Omnibus, Gitaly security projects. No Yes No
2 Adds workflow:: label to MRs. No No -
3 Creates QA issue. Yes No - The QA issue seems to be required for QA reports based on #1490 (comment 742851134). We can consider making it non-blocking as well.
4 Creates a Sentry deploy. No Yes No
5 Records/updates a deployment in https://ops.gitlab.net/gitlab-org/release/metadata. Yes Yes Yes

Items 3 and 5 are to be moved into separate rake tasks that can be called by independent blocking jobs. After that, we can make the tracking jobs non-blocking.

  1. Create separate rake tasks in release-tools to record a deployment in release/metadata, and to create a QA issue - gitlab-org/release-tools!1680 (merged).
  2. Create separate jobs in deployer that call the above rake tasks - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/458.
  3. Make the track jobs (in deployer) non-blocking for regular and rollback pipelines - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/462.

Cleanup:

  1. Cleanup the track_deployment rake task by removing the creation of the metadata deployment and QA issue - gitlab-org/release-tools!1688 (merged).
  2. Remove the no_metadata_tracking and no_qa variables from deployer - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/468.

Follow-up:

  • Make the create_qa_issue job non-blocking as well? We might be better off tackling this in a separate issue. create-qa-issue doesn't run in rollback pipelines though, so we might not need to worry about this.
Edited by Reuben Pereira