Extend the rollback logic to consider the post-deploy pipeline
Currently, when rolling back an auto-deploy package the rollback logic verifies if the package has post-migrations to determine if a rollback proceeds. With the introduction of the post-deploy migration pipeline, the rollback logic needs to consider if the post-migrations have been executed.
The successful executions of the post-deploy migration pipelines are tracked on the db/*
environments, establishing a point of no return for rollbacks, we need to adapt the rollback logic to consider this.
Proposal
Update the rollback logic to only allow rollbacks if the package is newer than the last post-deployment pipeline execution. Broadly, the implementation steps may look like this:
- Investigate which auto-deploy package was running when the post-deploy pipeline was executed. One idea to accomplish this is to fetch the last deployment on the
db/*
environments for theRelease::Metadata
project:
[3] pry(main)> deployment = client.last_successful_deployment(ReleaseTools::Project::Release::Metadata, 'db/gprd')
=> #<Gitlab::ObjectifiedHash:266940 {hash: {"id"=>352611, ...., "status"=>"success"}}
[4] pry(main)> product_version = ReleaseTools::ProductVersion.from_metadata_sha(deployment.sha)
2022-07-01 11:02:19.413568 I ReleaseTools::ProductVersion -- Fetching release metadata commit -- {:sha=>"5e16b1d136e4ea734661dba23169526cb1990a92"}
=> #<ReleaseTools::ProductVersion:0x00007fdd3c4ae630 @version="15.2.202206290920">
[5] pry(main)> product_version.auto_deploy_package
2022-07-01 11:02:28.460320 I ReleaseTools::ProductVersion -- Fetching release metadata -- {:version=>"15.2.202206290920"}
=> "15.2.202206290920-befec41d9ad.daa899cd5b3"
- If the package to rollback to is newer than
product_version.auto_deploy_package
we can rollback, if not, the rollback should not proceed. Example:
At the moment the package to rollback to (target
) is newer than the package that was tracked, so the rollback should proceed.
Implemented
The rollback check now checks on which version (minimum_version
) the post-deploy pipeline was last executed. It marks a rollback as available if the rollback target version is same or newer than the minimum_version
. The new logic checks the last successful deployment to the db/gprd
environment in the release/metadata
project to determine when the last post-deploy pipeline was executed.
This change has been implemented behind a feature flag called rollback_check_post_deploy_pipeline
: https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/224/edit.
Once we start using the post-deploy migration pipeline and stop using the post-deploy jobs in the coordinated pipeline, we can turn on the feature flag. We cannot turn on the feature flag while using the post-deploy jobs in the coordinated pipeline because the jobs in the coordinated pipeline do not create db/gstg
and db/gprd
deployments in the release/metadata
project.