Stage 3: Move post deploy migrations from being inside the deploy invocation of deployer to its own invocation of deployer
Currently post deploy migrations run inside deployer as part of the pipeline that deploys the code to an environment (non-canary stages).
In order to make it possible to move post-deploy migrations around, we want to keep post-deploy migrations inside the deployer pipeline, but split it from the deploy pipeline.
This means
- The pipeline invoked without
POSTDEPLOY_MIGRATIONS
set at all keeps the current state of things, so that chatops still works - With
POSTDEPLOY_MIGRATIONS
set tofalse
, the deployer pipeline will run but not do the post deploy migrations task - We need to make sure we can call a deployer pipeline with
RUN_POST_DEPLOY_MIGRATIONS
set totrue
and the ONLY task that is run out of the entire pipeline is the post-migrations for the pipeline in question - We need to then change the coordinated pipeline to call deployer twice, once with
RUN_POST_DEPLOY_MIGRATIONS
asfalse
thenRUN_POST_DEPLOY_MIGRATIONS
astrue
in the appropriate locations
Testing
-
Create a new branch with the CI configuration of ggillies/add-postdeploy-migrations-jobs
in ops https://ops.gitlab.net/gitlab-org/release/tools/-/tree/ggillies/add-postdeploy-migrations-jobs -
Protect the branch so it can reuse the protected environment variables. https://ops.gitlab.net/gitlab-org/release/tools/-/settings/repository -
Modify the tag
pipeline schedule to useggillies/add-postdeploy-migrations-jobs
https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/74/edit -
Tag coordinated pipeline based on the new branch (make sure to restore it back to master
, once the pipeline is created) -
Validate the following scenarios
General
-
Coordinated pipeline is successfully created - https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/974089
Post-migrations on staging
-
Deployment to staging is performed and it doesn't include the post-migration job https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/974328 -
Post-migrations bridge-job is triggered https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/974383 -
Post-migrations are executed https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/5922044 -
Successful notification is sent once the deployment, post-migrations, and QA are finished https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/5920295 -
Canary deployment starts once the successful notification is sent - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/974415
Post-migrations on gprd
-
Deployment to gprd is performed and it doesn't include the post-migration job https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/974602 -
Post-migrations bridge job is triggered - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/974753 -
Post-migrations are executed - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/5924291 -
Successful notification is sent once the deployment and post-migration job are finished - https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/5920314
Post-migration failure
- gstg
Failure notification Slack | Success notification after retry |
---|---|
![]() |
![]() |
Link | Link |
- gprd
Failure notification Slack | Success notification after retry |
---|---|
![]() |
![]() |
Link | Link |
Clean up
-
Unprotect branch in ops -
Remove branch from ops
What to do if something goes wrong?
- Change https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/74/edit to use
master
branch - Create a new coordinated pipeline
Status 12-01-2021
All the testing scenarios have been successfully tested. gitlab-org/release-tools!1630 (merged) has been merged. This issue is completed.
Edited by Mayra Cabrera