Manually mark migration as complete to fix deploy
Production Change
Change Summary
Originating Incident: #6642 (closed)
Manually update the schema version to allow reverts or deploys to continue.
Change Details
- Services Impacted - ServicePostgres
- Change Technician - @cmcfarland
- Change Reviewer - @skarbek
- Time tracking - Time, in minutes, needed to execute all change steps, including rollback
- Downtime Component - 0
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 1 minutes
-
Set label changein-progress on this issue
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 5 minutes
-
Log into the production leader database.
ssh patroni-v12-05-db-gprd.c.gitlab-production.internal
-
Start a TMUX session.
tmux
-
Verify you are on the leader.
sudo gitlab-patronictl list
-
Connect to the PostgreSQL console.
sudo gitlab-psql
-
Execute the SQL command.
insert into schema_migrations VALUES ('20220317161914');
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 10 minutes
-
Retry the migration job and it should pass. - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/6596152
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
There is no rollback, this mitigation is to help get the Deployer get past a stuck migration which will be handled in a future CR.
Monitoring
Key metrics to observe
- Metric: PostgreSQL Overview
- Location: https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1
- What changes to this metric should prompt a rollback:
Summary of infrastructure changes
- [ ] Does this change introduce new compute instances?
- [ ] Does this change re-size any existing compute instances?
- [ ] Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.
Edited by John Skarbek