Test and rollout the independent post-deploy pipeline
As part of &585 (closed), the behavior of the independent post-deploy pipeline needs to be thoroughly tested before introducing it into release activities.
Exit criteria
- Post-deploy migrations are executed on staging and production
- The post-deploy pipeline doesn't interfere with the staging and production deployments.
- Release managers are aware when the post-deploy pipeline is executed
- Evidence of the post-migrations executed through the post-deploy pipeline is available
- The execution of the post migrations is tracked across the database environments
Testing
Testing is based on the exit criteria and split into two phases:
- Phase 1: Initial testing to verify the configuration of the post-deploy pipeline
- Phase 2: Execution of post migrations in staging and production
- Phase 3: Rollout the post-deploy pipeline
✅ Phase 1: Initial testing
Details
Setup
-
Prepare release-tools to process and return fake post-migrations (this will prevent us waiting for actual post-migrations that can be time-consuming) gitlab-org/release-tools!1890 (merged) -
For each of these scenarios notify release managers
Tests
Without pending post-migrations
-
Ensure there are no pending post-migrations -
Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
-
Verify the prepare
job fails since there are no post-deploy migrations
With an ongoing staging deployment
-
Ensure there are no pending post-migrations -
Enable the fake_pending_post_migrations
feature flag -
Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
-
Verify the prepare
job is canceled and a message with theActiveGstgDeployment
is sent to Slack
With an ongoing production deployment
-
Enable the fake_pending_post_migrations
feature flag -
Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
-
Verify the prepare
job fails since there is an ongoing production deployment
With an active incident
-
Create a fake incident production#7264 (closed) -
Enable the fake_pending_post_migrations
feature flag -
Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
-
Verify the prepare
job fails
Lists the post-migrations to be executed
-
Ensure there are no pending post-migrations -
Enable the fake_pending_post_migrations
feature flag -
Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
-
Prevent post-migrations to be executed, e.g. cancel the jobs from the post_deploy_migrations:gstg
bridge. -
Verify a note has been added to release task issue with the list of pending post-migrations gitlab-org/release/tasks#3914 (comment 994680430) -
Add a comment to the note in the release task indicating it's part of the testing.
✅ Phase 2: Execution of post migrations in staging and production
Details
Tests
Standard execution for staging and production
Jun 28th details
Results on #2352 (comment 1009830702)
Setup:
-
Disable the fake_pending_post_migrations
feature flag -
Prepare an MR that creates a temporary table gitlab-org/gitlab!91119 (merged) -
Create a package that includes only that post-migration https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/1283271 -
Ensure there are pending post-migrations -
After the staging deployment cancel the post-migration job from the coordinated pipeline https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/7455319
For staging:
-
Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
-
Verify a slack message is sent to notify the execution of the post-migration pipeline -
Verify a message is posted in the release issue gitlab-org/release/tasks#4064 (comment 1009802310) -
Tracking -
Ensure a running
product deployment is created -
Verify a running
deployment is tracked in Canonical and Security fordb/gstg
-
-
Ensure the post-migrations are executed in staging. -
Verify a slack message is sent notifying the successful execution -
Tracking -
Verify a success
deployment is tracked in Canonical and Security -
Verify a success
product deployment is created -
Verify the merge requests environment have been updated with db/gstg
-
Verify the label workflow::post-deploy-db-staging
has been added.
-
For production:
-
Verify a slack message is sent to notify the execution of the post-migration pipeline -
Tracking -
Ensure a running
product deployment is created -
Verify a running
deployment is tracked in Canonical and Security fordb/gprd
-
-
Ensure the post-migrations are executed in production -
Verify a slack message is sent notifying the successful execution -
Tracking -
Verify a success
deployment is tracked in Canonical and Security -
Verify a success
product deployment is created -
Verify the merge requests environment have been updated with db/gprd
-
Verify the label workflow::post-deploy-db-production
has been added.
-
Clean up/rollback
-
Continue the execution of the coordinated pipeline
Setup:
-
Disable the fake_pending_post_migrations
feature flag -
Create a package that includes only that post-migration https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/1284818 -
Ensure there are pending post-migrations -
After the staging deployment cancel the post-migration job from the coordinated pipeline
For staging:
-
Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/1285550 -
Verify a slack message is sent to notify the execution of the post-migration pipeline -
Verify a message is posted in the release issue gitlab-org/release/tasks#4064 (comment 1011014044) -
Tracking https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/7462590 -
Ensure a running
product deployment is created -
Verify a running
deployment is tracked in Canonical and Security fordb/gstg
-
-
Ensure the post-migrations are executed in staging. https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/7462604 -
Verify a slack message is sent notifying the successful execution -
Tracking -
Verify a success
deployment is tracked in Canonical and Security -
Verify a success
product deployment is created -
Verify the merge requests environment have been updated with db/gstg
-
Verify the label workflow::post-deploy-db-staging
has been added.
-
For production:
-
Verify a slack message is sent to notify the execution of the post-migration pipeline -
Tracking https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/7462597 -
Ensure a running
product deployment is created -
Verify a running
deployment is tracked in Canonical and Security fordb/gprd
-
-
Ensure the post-migrations are executed in production https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/7462606 -
Verify a slack message is sent notifying the successful execution -
Tracking -
Verify a success
deployment is tracked in Canonical and Security -
Verify a success
product deployment is created -
Verify the merge requests environment have been updated with db/gprd
-
Verify the label workflow::post-deploy-db-production
has been added.
-
Clean up/rollback
-
Continue the execution of the coordinated pipeline
When a post-migration fails in staging and production
Setup:
-
Enable the fake_pending_post_migrations
feature flag -
Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/1295041
For staging:
-
Cancel the child job generated by the post_deploy_migrations:gstg
bridge job -
Verify a slack message is sent notifying the failed execution of the post migrations in staging -
Retry the child job from the post_deploy_migrations:gstg
bridge job -
Verify a successful slack message is sent
For production:
-
Cancel the child job generated by the post_deploy_migrations:gprd
bridge job -
Verify a slack message is sent notifying the failed execution of the post migrations in gprd -
Retry the child job from the post_deploy_migrations:gstg
bridge job -
Verify a successful slack message is sent
Quality is executed after the post-migrations are triggered on staging
Setup:
-
Enable fake_pending_post_migrations
feature flag -
Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/1295041
For staging:
-
Ensure the quality jobs are successfully triggered after the post-migrations are executed https://ops.gitlab.net/gitlab-org/quality/staging/-/pipelines/1295056
Phase 3: Rollout of the post-deploy pipeline
Preparation
-
Prepare a guide for the post-deploy pipeline. It should include execution instructions and troubleshoot section gitlab-org/release/docs!480 (merged) -
Prepare the announcement issue #2478 (closed). It should include: - What, why, when, and how is being introduced.
- The expectations for engineers and SRE's
- Roll out process and dates
-
Complete Prepare the coordinated pipeline for the post-deploy migration pipeline transition by merging gitlab-org/release-tools!1919 (merged) -
Enable the rollback_check_post_deploy_pipeline -
Add a slack reminder to remind release managers about the execution of the post deploy pipeline. => Reminders set every weekday 12UTC and 23UTC -
Make the public announcement of the rollout -
Ensure the Delivery team is aware of it -
Ensure Release managers are aware of it -
Ping database maintainers in the announcement issue for awareness -
Share the announcement in the Engineering Week in Review doc -
Share in development channels.
-
Rollout
-
Ensure the post-deploy pipeline is executed at least once a day by EMEA or AMER release manager. -
Observe the behavior.
If a week has passed with no major problems, the rollout has been completed.
Handling post-migrations failures.
- If the post migration execution failed on staging or production, follow the handling post migration failure guideline
- If post migrations are not being executed by the post-deploy pipeline, follow the post migrations not being executed guideline
Clean up
After the testing has been completed:
-
Revert gitlab-org/release-tools!1890 (merged) => gitlab-org/release-tools!1941 (merged) -
Remove the fake_pending_post_migrations
feature flag -
If temporary tables were created ensure they're reverted gitlab-org/gitlab!91119 (merged) => gitlab-org/gitlab!92571 (diffs) -
Broadcast the rollout of the post-deploy pipeline has been finished -
Share the announcement in the Engineering Week in Review doc -
Share in development channels. -
Ensure upcoming release managers are aware of it https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1221#note_1035919919
-