Reorder deployments bigint cleanup so the FK sorts first
This merge request swaps the timestamp prefixes of two post-deployment migrations in the 19.3 milestone so the temporary foreign key removal on deployments runs before the removal of the indexes supporting it. It is a file rename only: class names and migration bodies are unchanged. The previous order violated the documented rule for dropping foreign keys and caused an incident in which project and group deletions were blocked.
What changed
| Migration | Before | After |
|---|---|---|
RemoveTmpBigintFkForDeploymentsPhaseTwoRetryTwo (removes the foreign key) |
20260811082241 |
20260731085558 |
DropTmpBigintIndexesAndFkForDeploymentsPhaseTwo (removes the temporary indexes) |
20260731085558 |
20260811082241 |
The two empty migrations, 20260805084521 RemoveTmpBigintFkForDeploymentsPhaseTwo and 20260810085131 RemoveTmpBigintFkForDeploymentsPhaseTwoRetry, keep their timestamps and are unaffected.
Why
The deployments table went through an integer to bigint conversion for project_id and user_id. That conversion created temporary indexes on deployments.project_id_convert_to_bigint and a temporary foreign key fk_b9a3851b82_tmp on that column referencing projects(id) with ON DELETE CASCADE. Cleaning those up has an order requirement, from doc/development/database/foreign_keys.md:
foreign keys must be removed before removing indexes supporting these foreign keys
The same section explains why:
Without an index on the foreign key it forces Postgres to do a full table scan every time a record is deleted from the referenced table. In the past this has led to incidents where deleting
projectsandnamespacestimes out.
In production the index removals succeeded and the foreign key removal did not take effect. Project and group deletions were then blocked by full table scans on deployments.
Migration ordering after this change
20260731085558RemoveTmpBigintFkForDeploymentsPhaseTwoRetryTwo: removes the foreign key, using escalating lock timeouts20260805084521RemoveTmpBigintFkForDeploymentsPhaseTwo: empty body20260810085131RemoveTmpBigintFkForDeploymentsPhaseTwoRetry: empty body20260811082241DropTmpBigintIndexesAndFkForDeploymentsPhaseTwo: removes the temporary indexes
The two empty migrations were the original foreign key removal and its first retry. Both failed in production and were emptied out afterward, because a recorded migration version never runs again. They are no-ops, so their position between the foreign key removal and the index removal does not matter.
DropTmpBigintIndexesAndFkForDeploymentsPhaseTwo still carries "AndFk" in its class name even though the foreign key logic moved out of it earlier. It is deliberately not renamed, to keep the diff to a rename.
Impact
Every installation that already ran these migrations, including GitLab.com, has all four versions recorded, so none of them run again and nothing changes there. The reorder corrects the sequence for installations that run the 19.3 migrations from this point forward, and removes a wrong example from the codebase.
Verification
- The migration checksum checker reports no issues.
- No
db/schema_migrations/checksum files change: all four timestamps already exist and are reused rather than created. - No
db/structure.sqlchange, since it carries no migration versions. - Rubocop passes on both renamed files.
References
- gitlab-com/gl-infra/production-engineering#29590 (closed) (corrective action work item)
- !248030 (merged) (added the index cleanup)
- !248679 (merged) (split the foreign key removal into its own migration)
- !249509 (merged) (retry with longer lock timeouts)
- #551602 (the bigint conversion)