Database Migrations unsafe for auto-deploy
Problem Statement
We recently had a database migration fail on one of our test sandboxes. The cause of the failure is due to a problem with the migration itself. This exposed an issue with the way the Dedicated tooling currently processes database migrations. There's a few issues that we must address prior to enabling auto-deploy into Cells:
- Database migrations need to be better timed - currently these are being executed after various upgrades which are non-compliant with the recommended upgrade strategy
- Nothing validates that migrations are applied - this is a blind strategy and we assume success with the caveat the job must succeed
An example of how this was detected in our auto-deployed Delivery Tenant: https://ops.gitlab.net/gitlab-com/delivery/cells-tissue/-/jobs/13309773
Solutions
-
Investigate -
Determine how we can ensure migrations are up-to-date for all deployments
Edited by John Skarbek