Add guidance on reverting database migration change in "broken master" process
Why is this change being made?
We recently had a broken master incident (gitlab-org/gitlab#387761 (closed)) that needed to undo the database migration executed in staging and block the deployment to prevent the migration from running on production. I reverted the MR that introduced the migration, but learned that the revert was insufficient from @tkuah
's below comment:
Quoting the comment from @tkuah
in gitlab-org/gitlab!108644 (comment 1235023691):
FYI I have opened gitlab-com/gl-infra/production#8229 (closed) because a revert is not sufficient - gitlab-org/gitlab!104155 (merged) might be deployed; and then another deploy that includes this MR. This will then make GitLab.com DB schema inconsistent with everything else
This MR is to update the master-broken documentation to reflect the additional step that must be taken when reverting database migration changes.
Author Checklist
-
Provided a concise title for this Merge Request (MR) -
Added a description to this MR explaining the reasons for the proposed change, per say why, not just what - Copy/paste the Slack conversation to document it for later, or upload screenshots. Verify that no confidential data is added.
-
Assign reviewers for this MR to the correct Directly Responsible Individual/s (DRI) - If the DRI for the page/s being updated isn’t immediately clear, then assign it to one of the people listed in the
Maintained by
section on the page being edited - If your manager does not have merge rights, please ask someone to merge it AFTER it has been approved by your manager in #mr-buddies
- If the DRI for the page/s being updated isn’t immediately clear, then assign it to one of the people listed in the
-
If the changes affect team members, or warrant an announcement in another way, please consider posting an update in #whats-happening-at-gitlab linking to this MR - If this is a change that directly impacts the majority of global team members, it should be a candidate for #company-fyi. Please work with internal communications and check the handbook for examples.