Add CI job to verify execution of migrations in two steps
What does this MR do and why?
We had a recent incident (gitlab-com/gl-infra/production#7762 (closed)), where the root cause was that during development and test we don't execute database migrations the same way we do when deploying to production.
In development and test we usually execute them all together, while during deployment we execute regular migrations first, and post-deployment migrations some time later at the discretion of release managers.
This can lead to incidents like the one above - because of the order of migrations change, post-deployment migration may fail.
This MR adds a new CI job that executed migrations in two steps, same as what we do when deploying. This should catch issues like this earlier.
Related to gitlab-com/gl-infra/production#7762 (comment 1108028695).
How to set up and validate locally
Example MR to verify the job - !99058 (closed).
It adds 3 new migrations:
- Regular migration that creates a table
- Post-deployment migration that drops a column from the same table
- regular migration that drops the table
When executed together it works OK, but when executed in two steps, the post-deployment migration fails, as the table it tries to change is no longer present. This is caught with in the new job - https://gitlab.com/gitlab-org/gitlab/-/jobs/3116187426#L1117.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.