<!-- Please review https://about.gitlab.com/handbook/engineering/infrastructure/change-management/ for the most recent information on our change plans and execution policies. --> # Production Change ### Change Summary Originating Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6642 Manually update the schema version to allow reverts or deploys to continue. ### Change Details 1. **Services Impacted** - ~"Service::Postgres" 1. **Change Technician** - <!-- woodhouse: '`@{{ .Username }}`' -->@cmcfarland 1. **Change Reviewer** - @skarbek 1. **Time tracking** - <!-- woodhouse: '{{ .Duration}}' -->{+ Time, in minutes, needed to execute all change steps, including rollback +} 1. **Downtime Component** - 0 ## Detailed steps for the change ### Pre-Change Steps - steps to be completed before execution of the change *Estimated Time to Complete (mins)* - 1 minutes - [x] Set label ~"change::in-progress" on this issue ### Change Steps - steps to take to execute the change *Estimated Time to Complete (mins)* - 5 minutes - [x] Log into the production leader database. ``` ssh patroni-v12-05-db-gprd.c.gitlab-production.internal ``` - [x] Start a TMUX session. ``` tmux ``` - [x] Verify you are on the leader. ``` sudo gitlab-patronictl list ``` - [x] Connect to the PostgreSQL console. ``` sudo gitlab-psql ``` - [x] Execute the SQL command. ``` insert into schema_migrations VALUES ('20220317161914'); ``` ### Post-Change Steps - steps to take to verify the change *Estimated Time to Complete (mins)* - 10 minutes - [x] Retry the migration job and it should pass. - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/6596152 ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change There is no rollback, this mitigation is to help get the Deployer get past a stuck migration which will be handled in a future CR. ## Monitoring ### Key metrics to observe <!-- * Describe which dashboards and which specific metrics we should be monitoring related to this change using the format below. --> - Metric: PostgreSQL Overview - Location: https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1 - What changes to this metric should prompt a rollback: ## Summary of infrastructure changes ~~- [ ] Does this change introduce new compute instances?~~ ~~- [ ] Does this change re-size any existing compute instances?~~ ~~- [ ] Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?~~ <!-- * If you answer yes to any of the items in this checklist, summarize below. --> {+Summary of the above+} ## Change Reviewer checklist <!-- To be filled out by the reviewer. --> ~C4 ~C3 ~C2 ~C1: - [ ] The **scheduled day and time** of execution of the change is appropriate. - [ ] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [ ] The change plan includes **estimated timing values** based on previous testing. - [ ] The change plan includes a viable [rollback plan](#rollback). - [ ] The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [ ] The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - [ ] The change plan includes success measures for all steps/milestones during the execution. - [ ] The change adequately minimizes risk within the environment/service. - [ ] The performance implications of executing the change are well-understood and documented. - [ ] The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - [ ] The change has a primary and secondary SRE with knowledge of the details available during the change window. ## Change Technician checklist <!-- To find out who is on-call, in #production channel run: /chatops run oncall production. --> - [x] This issue has a criticality label (e.g. ~C1, ~C2, ~C3, ~C4) and a change-type label (e.g. ~"change::unscheduled", ~"change::scheduled") based on the [Change Management Criticalities](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-criticalities). - [x] This issue has the change technician as the assignee. - [x] Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [ ] Necessary approvals have been completed based on the [Change Management Workflow](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows). - [ ] Change has been tested in staging and results noted in a comment on this issue. - [ ] A dry-run has been conducted and results noted in a comment on this issue. - [x] SRE on-call has been informed prior to change being rolled out. (In #production channel, mention `@sre-oncall` and this issue and await their acknowledgement.) - [ ] Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=Incident%3A%3AActive).