Announcement: The post-deploy migrations execution is changing to be independent of the auto-deploy process
What is changing?
The process for executing post-deploy migrations on staging and production on GitLab.com is changing. Currently, the post-deploy migrations are executed at the end of the auto-deploy process, (once the deployment to staging and production has been completed) after this change, post-deploy migrations will run once a day on a schedule separated from the auto-deploy process. If you are a part of Engineering this could impact your workflow
Auto-deploy process (current) | Auto-deploy process (after) |
---|---|
![]() |
![]() |
How will the post-deploy migrations be executed?
As part of &585 (closed), a new post-deploy migration (PDM) pipeline independent from the GitLab auto-deploy process will be introduced. The sole purpose of this pipeline is to execute post-deploy migrations in the staging and production environments.
For starters, the PDM pipeline is expected to be triggered at least once a day at the discretion of release managers. Note the execution of this new pipeline respects the same change lock periods as the auto-deploy pipeline: It can be executed from Monday to Friday and will be skipped during production change locks.
When will the transition take place?
The testing and rollout of the PDM will be done during the %15.2 development cycle and completed before the %15.2 release. Right now we're in the final stage of testing, running against staging and production environments with the ability to fallback to the previous system if needed.
Following full testing, we'll complete the transition and announce in development channels as well as Engineering-week-in-review.
Update - 2022-07-21
Rollout and full testing of the PDM have been completed. The pipeline has been incorporated into the daily and monthrly release activities as the principal way to execute post migrations.
Why are we changing the execution of the post-deploy migrations?
Post-deploy migrations are blockers for rollbacks, they can’t be rolled back due to the nature of their operations, because of this, any package deployed with post-deployment migrations prevents a rollback regardless of the incident cause. This impacts the availability of GitLab.com.
As an engineer how does this impact me?
If you are a part of Engineering this could impact your workflow. When working on a feature that depends on a post-deploy migration being executed on GitLab.com, consider the execution of it will no longer be tied to a production deployment. Instead, barring any production blockers, the post-deploy migrations will be executed at least once a day and ahead of any releases for self-managed users.
The process for merging a post-migration into the GitLab codebase stays the same.
How can I determine if a post-deploy migration has been executed?
Same as determining the status of a merged merge request. The status of a post-deploy migration will be visible in the merge request widget and labels:
- The merge request widget will reflect the db environments,
db/gstg
for staging anddb/gprd
for production, once the post-deploy migration has been executed. - The merge request labels will be updated with
workflow::post-deploy-db-staging
andworkflow::post-deploy-db-production
when the post-deploy migration has been executed.
Widget | Labels |
---|---|
![]() |
![]() |
As an EOC how does this impact me?
Post-deploy migrations execution is no longer tied to production deployments, regardless, incidents associated with post-deploy migrations failures may still be treated as production incidents (see the post-deploy migration failure guide).
Information about the execution of the post-deploy migrations can be found on:
- The #announcements channel. The PDM pipeline will notify the start, end, and status of the execution of each environment - example.
- The monthly release issue. A message is posted at the beginning of the PDM pipeline listing the post-deploy migrations to be executed - example.
There are plans to make the post-deploy migration information more accessible (#587 and #2417). Feedback welcome.
I noticed a post-deploy migration hasn’t been executed, what should I do?
If it has been over 24h since the merge request with a post-deploy migration was deployed to GitLab.com and the environment widget nor the labels have been updated, please post a comment on this issue or ping the release managers in the #releases channel.
Where can I read more about the new PDM pipeline?
More information can be seen on:
- Epic - Remove the blocking nature of post-deploy migrations.
- Overview of the PDM guideline.
- Testing and rollout of the PDM pipeline
Where can I ask questions or provide feedback?
Please comment on this issue directly with your questions and feedback and we can provide answers there. If it is urgent please reach out to #releases in slack.