<!--Please review https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/ for the most recent information on our change plans and execution policies.--> # Production Change ## Change Summary > [!important] > > This change is a prerequisite to upgrading our Ops instance to the next major version of GitLab. We need to upgrade `ops.gitlab.net` to **GitLab v19** to keep it in line with our production GitLab instance, however it [requires PostgreSQL 17 minimum](https://docs.gitlab.com/update/versions/gitlab_19_changes/#postgresql-17-minimum-requirement). **Post-incident context:** following yesterday's [incident #13499](https://app.incident.io/gitlab/incidents/13499) and the resulting database data loss, a replacement Cloud SQL instance (**`ops-central-2`**) was stood up in **`us-east1`** (because `us-central1` had no capacity at the time). So `ops.gitlab.net` currently runs its application in `us-central1` while its database lives in `us-east1` (`ops-central-2`), a cross-region split we want to undo. This change therefore happens in **two phases**: 1. **Phase 1: migrate the database back to `us-central1`.** Snapshot `ops-central-2` (us-east1), restore it into **`ops-central`** (us-central1), repoint `ops.gitlab.net` at `ops-central`, verify, and take a fresh post-migration snapshot. 2. **Phase 2: upgrade `ops-central` to PostgreSQL 17.** In-place major version upgrade of `ops-central` from PostgreSQL 16 to 17. The PostgreSQL 17 upgrade is done **in-place via the Google Cloud Console** (not via Terraform), to avoid the chicken-and-egg situation of the ops instance being in maintenance mode and therefore unable to apply a Terraform change to a project (`config-mgmt`) hosted on that same instance. Config-mgmt [MR 15169](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15169) (`database_version` 16 → 17) is applied via Atlantis and merged **after** maintenance mode is lifted, purely to bring the Terraform state back in line with reality. The `ops-central` instance is `ENTERPRISE_PLUS`, which minimises downtime during upgrades. The previous major version upgrade of this same instance (v14 → v16, see [production#20074](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20074)) observed near-zero downtime (< 1 second). > [!note] > > **A short burst of 5XX from `ops.gitlab.net` is possible** during the DB repoint (Phase 1) and while the instance restarts during the upgrade (Phase 2). The dataset is small (**~210 GiB**), so any impact is expected to be brief (**at most a few minutes**), on top of the maintenance-mode page shown to users. Maintenance mode is enabled as a precaution in both phases. > [!warning] > > **ArgoCD for the ops cluster reads from a mirror on `ops.gitlab.net`** (the `argocd-apps` Application's `repoURL` is `https://ops.gitlab.net/gitlab-com/gl-infra/argocd/apps.git`, push-mirrored from gitlab.com). While maintenance mode is on, that mirror push is blocked, so a change merged on gitlab.com in-window would not reach ArgoCD. To avoid that, the Phase 1 repoint MR is **merged and mirrored ahead of the window**, with **autosync disabled**, so it does not roll out until we trigger a **manual sync** at the right moment. Because the commit is already on the mirror, that manual sync works even while ops.gitlab.net is in maintenance mode (no new push is needed). > > Disabling autosync must be done by **patching the live `gitlab-ops--<hash>` Application directly** (`kubectl patch`), **not** via a `service.yaml` change. Verified in the `argocd-generic-service` chart (v0.7.0, `gitlab/argocd-generic-service/templates/applicationset.yaml`): every generated service ApplicationSet hardcodes `ignoreApplicationDifferences` on `/spec/syncPolicy/automated/enabled` (and the `gitlab.com/pause-sync` label). That ignore rule cuts both ways: the controller neither propagates a `service.yaml`/template change of that field to the already-generated app, nor reverts a manual change to it. So the direct patch is authoritative, and the ignore rule then keeps it disabled. (This matches the Redis to Valkey migration observation, [production#22684](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22684).) Change criticality: **~C2**: planned maintenance window on the internal ops instance, involving a database region migration (restore) followed by a major version upgrade of its primary database. Issue: https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29613 ## Change Details <!--To automatically add your change to the GitLab Production calendar update the following fields: - Time tracking - Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM) Bot: https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler--> 1. **Services Impacted** - ~"Service::OpsGitlabNet" ~"Service::CloudSQL" 2. **Change Technician** - `@gsgl` 3. **Change Reviewer** - @jomartin1 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-08-28 00:00 5. **Time tracking** - 120 6. **Downtime Component** - Two short maintenance windows. Phase 1 (DB region migration) requires `ops.gitlab.net` in maintenance mode while the DB is restored and repointed (dominated by the restore, ~12 min). Phase 2 (PG17 upgrade) is expected near-zero (`ENTERPRISE_PLUS`; the previous v14 → v16 upgrade of this instance observed < 1s), with a possible brief 5XX burst while the instance restarts. Maintenance mode is enabled as a precaution in both phases. ### Set Maintenance Mode in GitLab This change involves scheduled maintenance. Maintenance mode is set and unset as part of the steps below so SLA calculations adjust for the maintenance period ([runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md)). ## Detailed steps for the change ### Pre-execution steps - [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done. - [x] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. (In `#production` channel, mention `@sre-oncall` and this issue and await their acknowledgement.) - [x] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [x] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2. - [x] An appropriate alert silence targeting the ops environment is added for the duration of the change. - [x] **`ops-central` exists and is running in `us-central1`**, so it is a valid restore target for the `gcloud sql backups restore` in Phase 1. Confirm it is PostgreSQL 16 (the same version as `ops-central-2`) so the restore is version-compatible. - [x] **Disable ArgoCD autosync for `gitlab-ops` by patching the live Application directly**, and confirm it reads `false`. A `service.yaml` autosync change will **not** take effect on the already-generated app: the `argocd-generic-service` chart (v0.7.0) hardcodes `ignoreApplicationDifferences` on `/spec/syncPolicy/automated/enabled`, so template changes to that field are ignored (and manual changes are preserved): ```bash APP="$(kubectl -n argocd get applications.argoproj.io -o name | grep -m1 gitlab-ops | cut -d/ -f2)" kubectl -n argocd patch application "$APP" --type merge -p '{"spec":{"syncPolicy":{"automated":{"enabled":false}}}}' kubectl -n argocd get application "$APP" -o jsonpath='{.spec.syncPolicy.automated.enabled}' # must print: false ``` - [x] **Merge the repoint MR ahead of the window** ([argocd/apps!3259](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3259)) and confirm it has mirrored to `ops.gitlab.net`. It is a **Draft**; do not mark it ready and merge until autosync is confirmed off (previous step). With autosync off, this does not roll out; pre-staging it on the mirror is what lets the in-window manual sync work while ops.gitlab.net is in maintenance mode. - [x] Confirm config-mgmt [MR 15169](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15169) is reviewed and approved, ready to apply via Atlantis after Phase 2. ### Change steps - steps to take to execute the change *Estimated Time to Complete (mins)* - 120 minutes - [x] Set label ~"change::in-progress" `/label ~change::in-progress`. - [x] Silence [CiRunnersServicePollingErrorSLOViolation for ops](https://alerts.gitlab.net/#/silences/new?filter=%7Baggregation%3D%22component%22%2C%20alert_class%3D%22slo_violation%22%2C%20alert_type%3D%22symptom%22%2C%20component%3D%22polling%22%2C%20env%3D%22ops%22%2C%20environment%3D%22ops%22%2C%20feature_category%3D%22runner%22%2C%20monitor%3D%22global%22%2C%20pager%3D%22pagerduty%22%2C%20rules_domain%3D%22general%22%2C%20severity%3D%22s2%22%2C%20sli_type%3D%22error%22%2C%20slo_alert%3D%22yes%22%2C%20stage%3D%22main%22%2C%20tier%3D%22runners%22%2C%20type%3D%22ci-runners%22%2C%20user_impacting%3D%22yes%22%2C%20window%3D%221h%22%2C%20alertname%3D%7E%22CiRunnersServicePollingErrorSLOViolation%22%7D). - [x] Silence the [ops.gitlab.net sign-in page blackbox probe (`BlackboxProbeFailuresLong`)](https://alerts.gitlab.net/#/silences/new?filter=%7Balert_type%3D%22symptom%22%2C%20cluster%3D%22gprd-gitlab-gke%22%2C%20env%3D%22gprd%22%2C%20environment%3D%22gprd%22%2C%20instance%3D%22https://ops.gitlab.net/users/sign_in%22%2C%20job%3D%22scrapeConfig/monitoring/prometheus-agent-blackbox%22%2C%20module%3D%22http_2xx%22%2C%20monitor%3D%22default%22%2C%20pager%3D%22pagerduty%22%2C%20prometheus%3D%22monitoring/gitlab-rw-prometheus%22%2C%20provider%3D%22gcp%22%2C%20region%3D%22us-east1%22%2C%20severity%3D%22s1%22%2C%20shard%3D%22default%22%2C%20stage%3D%22main%22%2C%20tier%3D%22sv%22%2C%20type%3D%22blackbox%22%2C%20alertname%3D%7E%22BlackboxProbeFailuresLong%22%7D), **set to expire after 2 hours**. The sign-in page will be unavailable while ops.gitlab.net is in maintenance mode and during the DB restore/upgrade, which would otherwise page (this probe is `severity::1`). #### Phase 1: Migrate the database back to `us-central1` (`ops-central`) - [x] **Confirm ArgoCD autosync for `gitlab-ops` is OFF** (disabled ahead of the window; see pre-execution) and that the repoint MR has already mirrored to `ops.gitlab.net`. Re-check the live Application, since a template change would not have propagated: ```bash APP="$(kubectl -n argocd get applications.argoproj.io -o name | grep -m1 gitlab-ops | cut -d/ -f2)" kubectl -n argocd get application "$APP" -o jsonpath='{.spec.syncPolicy.automated.enabled}' # must print: false ``` - [x] **Put GitLab into maintenance mode** \[[instructions](https://docs.gitlab.com/administration/maintenance_mode/#enable-maintenance-mode)\]: log in to the ops instance with the ops-contact admin credentials, then Admin \> Settings \> General \> toggle `Enable Maintenance Mode` to true, and **Save changes**. - [x] **Scale Sidekiq to 0.** No need to touch the HPA: an HPA will not scale a Deployment that is already at 0 replicas (`scaling is disabled since the replica count of the target is zero`), so scaling to 0 holds (confirmed during the Redis to Valkey migration). With autosync also off, ArgoCD will not restore it either. ```bash kubectl -n gitlab scale deploy ops-gitlab-sidekiq-all-in-1-v2 --replicas=0 ``` Confirm no active Sidekiq jobs remain before snapshotting. - [x] **Take a snapshot of `ops-central-2`** (the current us-east1 DB) and note the backup ID: ```bash gcloud sql backups create --instance=ops-central-2 --project=gitlab-ops \ --description="pre-migration snapshot: ops-central-2 (us-east1) -> ops-central (us-central1)" gcloud sql backups list --instance=ops-central-2 --project=gitlab-ops # note the BACKUP ID ``` - [x] **Restore that snapshot into `ops-central`** (us-central1). This overwrites `ops-central` with the `ops-central-2` data: ```bash gcloud sql backups restore <BACKUP ID> \ --restore-instance=ops-central \ --backup-instance=ops-central-2 \ --project=gitlab-ops ``` Wait for the restore to finish and for `ops-central` to return to `RUNNABLE` (up to ~12 min). - [x] **Manually sync `gitlab-ops` in the ArgoCD UI** to roll out the repoint at `ops-central` (select the `gitlab-ops--<hash>` app and click **Sync**). The repoint MR ([argocd/apps!3259](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3259)) was merged and mirrored ahead of the window and autosync is off, so nothing changed until now; the manual sync applies the already-mirrored commit (which works even in maintenance mode, since no new push to ops.gitlab.net is needed). Roll the GitLab deployments if needed so pods reconnect to `ops-central`. - [x] **Scale Sidekiq back to 3.** The HPA will **not** scale a Deployment up from 0 (`scaling is disabled since the replica count of the target is zero`), so scale manually to `minReplicas`; the HPA then takes over: ```bash kubectl -n gitlab scale deploy ops-gitlab-sidekiq-all-in-1-v2 --replicas=3 kubectl -n gitlab get hpa ops-gitlab-sidekiq-all-in-1-v2 # confirm ScalingActive=True ``` - [x] **Remove GitLab from maintenance mode** (Admin \> Settings \> General \> toggle `Enable Maintenance Mode` to false, **Save changes**). - [x] **Smoke test** that essential functionality works on `ops-central`: web UI loads, MRs/issues render (DB reads), `git pull`/`git push` work, a CI pipeline runs, and there are no DB connection errors in the logs. - [x] **Take a post-migration snapshot of `ops-central`** (clean rollback point before the upgrade): ```bash gcloud sql backups create --instance=ops-central --project=gitlab-ops \ --description="post-migration to us-central1, pre-PG17" ``` #### Phase 2: Upgrade `ops-central` to PostgreSQL 17 - [x] Connect to the database via the toolbox pod and open `psql` (resolve the pod dynamically; password read from `/srv/gitlab/config/database.yml` → `production.main.password`, so no prompt). Run the `ANALYZE` checks below in this session: ```bash # connect to the ops-central cluster glsh kube use-cluster ops-central # find the toolbox pod by label, then open psql through it (password sourced from database.yml) TOOLBOX_POD="$(kubectl -n gitlab get pod -l app=toolbox,release=ops-gitlab -o jsonpath='{.items[0].metadata.name}')" kubectl -n gitlab exec -ti "$TOOLBOX_POD" -c toolbox -- bash -lc \ 'PGPASSWORD="$(ruby -ryaml -e "puts YAML.load_file(%q{/srv/gitlab/config/database.yml})[%q{production}][%q{main}][%q{password}]")" psql -h 127.0.0.1 -p 5432 -U gitlab gitlabhq_production' ``` - [x] Determine current state of `ANALYZE`: ```sql select schemaname,relname,last_vacuum,last_autovacuum,last_analyze,last_autoanalyze from pg_stat_user_tables order by last_analyze asc nulls last; ``` - expect: lots of recent GitLab-performed analyze operations for partitions and partitioned tables, notably CI. - [x] Test analyze works with available access: ```sql SET statement_timeout = 0; ANALYZE (VERBOSE, BUFFER_USAGE_LIMIT "1 GB"); SET statement_timeout TO DEFAULT; ``` - [x] Verify current state of `ANALYZE`: ```sql select schemaname,relname,last_vacuum,last_autovacuum,last_analyze,last_autoanalyze from pg_stat_user_tables order by last_analyze desc nulls first; ``` - [x] Put GitLab into maintenance mode again \[[instructions](https://docs.gitlab.com/administration/maintenance_mode/#enable-maintenance-mode)\]: - [x] Log in to the ops instance using the ops-contact admin credentials. - [x] Admin \> Settings \> General \> toggle `Enable Maintenance Mode` to true, and **Save changes**. - [x] **Scale Sidekiq to 0** so no jobs run against the database during the upgrade. Scaling to 0 holds (the HPA will not scale a 0-replica target back up, and autosync is still off), and confirm no active jobs remain: ```bash kubectl -n gitlab scale deploy ops-gitlab-sidekiq-all-in-1-v2 --replicas=0 ``` - [x] **Upgrade the `ops-central` instance to PostgreSQL 17 via the [Google Cloud Console](https://console.cloud.google.com/sql/instances/ops-central/edit?project=gitlab-ops).** - [x] This is done in the Console (not Terraform) to avoid the chicken-and-egg situation of the ops instance being in maintenance mode and unable to apply the Terraform change to a project hosted on that instance. - [x] Cloud SQL automatically takes a **pre-upgrade backup** (POSTGRES_16 → POSTGRES_17) before starting; this is the rollback point. - [x] Validate the instance has been updated to **PostgreSQL 17** (`gcloud sql instances describe ops-central --project=gitlab-ops --format='value(databaseVersion,state)'` → `POSTGRES_17` / `RUNNABLE`), and that the HA standby is healthy. - [x] Run `ANALYZE` on the newly upgraded CloudSQL instance (a major version upgrade resets planner statistics): - [x] Connect via `kubectl`, resolving the toolbox pod dynamically (no hardcoded pod name); password read from `/srv/gitlab/config/database.yml` → `production.main.password`, so no prompt: ```bash # connect to the ops-central cluster glsh kube use-cluster ops-central # find the toolbox pod by label, then open psql through it (password sourced from database.yml) TOOLBOX_POD="$(kubectl -n gitlab get pod -l app=toolbox,release=ops-gitlab -o jsonpath='{.items[0].metadata.name}')" kubectl -n gitlab exec -ti "$TOOLBOX_POD" -c toolbox -- bash -lc \ 'PGPASSWORD="$(ruby -ryaml -e "puts YAML.load_file(%q{/srv/gitlab/config/database.yml})[%q{production}][%q{main}][%q{password}]")" psql -h 127.0.0.1 -p 5432 -U gitlab gitlabhq_production' ``` - [x] Run analyze: ```sql analyze; ``` - [x] Check analyze ran correctly: ```sql SELECT schemaname, relname AS table_name, last_analyze, last_autoanalyze FROM pg_stat_all_tables WHERE last_analyze IS NOT NULL OR last_autoanalyze IS NOT NULL ORDER BY greatest(COALESCE(last_analyze, '1970-01-01'), COALESCE(last_autoanalyze, '1970-01-01')) DESC; ``` - [x] **Scale Sidekiq back to 3.** The HPA will not scale a Deployment up from 0, so scale manually to `minReplicas`; the HPA then takes over: ```bash kubectl -n gitlab scale deploy ops-gitlab-sidekiq-all-in-1-v2 --replicas=3 kubectl -n gitlab get hpa ops-gitlab-sidekiq-all-in-1-v2 # confirm ScalingActive=True ``` - [x] Final validation that everything is working as expected (web UI loads, MRs/issues render, no DB connection errors in the logs). - [x] Remove GitLab from maintenance mode \[[instructions](https://docs.gitlab.com/administration/maintenance_mode/#enable-maintenance-mode)\]: - [x] Log in to the ops instance using the ops-contact admin credentials. - [x] Admin \> Settings \> General \> toggle `Enable Maintenance Mode` to false, and **Save changes**. - [x] Apply config-mgmt [MR 15169](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15169) to bring the Terraform state back in line with reality (the instance is already PG17 from the Console upgrade). Do this promptly, and do not run any other ops-environment apply from `main` in the meantime (config on `main` still says `POSTGRES_16`, so an apply from `main` would attempt an unsupported downgrade): - [x] Run an updated `atlantis plan` and confirm the plan looks as expected (should be a no-op / state reconciliation, not a change to the instance). - [x] Run `atlantis apply -p ops`. - [x] Merge the MR. #### Wrap-up - [x] **Re-enable ArgoCD autosync** on the Application. Git already reflects the repoint (the MR was merged ahead of the window), so live and desired match: `kubectl -n argocd patch application "$APP" --type merge -p '{"spec":{"syncPolicy":{"automated":{"enabled":true}}}}'`. Confirm the app is `Synced`/`Healthy` with no diff. - [x] Remove alert silences. - [x] Follow-up (out of window): decommission the `ops-central-2` (us-east1) instance once `ops-central` is confirmed good. - [x] Set label ~"change::complete" `/label ~change::complete`. > [!tip] > > CloudSQL runbook: [Related CloudSQL Runbook with Rough Upgrade Process](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/cloudsql/cloudsql.md) ## Rollback > [!caution] > > A rollback should **only be done** if maintenance mode is still turned on. ### Phase 1 (DB migration) rollback `ops-central-2` (us-east1) is left untouched during Phase 1, so it remains a warm rollback target with the current data. - [ ] Point `gitlab-ops` back at `ops-central-2` (us-east1). If the repoint has not been manually synced yet, there is nothing to undo (the app is still on `ops-central-2`). If it was already synced, in the ArgoCD UI **Sync** `gitlab-ops` to the pre-repoint git revision (already on the mirror) by setting the revision to the commit before the repoint, so `global.cloudSQL.region` / instance revert to `us-east1` / `ops-central-2`. - [ ] Scale Sidekiq back to `minReplicas` (`kubectl -n gitlab scale deploy ops-gitlab-sidekiq-all-in-1-v2 --replicas=3`). - [ ] Confirm `ops.gitlab.net` is serving from `ops-central-2` again, then lift maintenance mode. - [ ] Set label ~"change::aborted" `/label ~change::aborted`. ### Phase 2 (PG17 upgrade) rollback - [ ] Create a new CloudSQL instance with PostgreSQL 16 (the pre-upgrade version). It takes approx 15 min to spin up an empty CloudSQL instance. (Alternatively, restore into the post-migration snapshot target.) - [ ] Restore the pre-upgrade backup to the new/target instance (takes up to ~12 min to restore): - [ ] Go to [CloudSQL backups for ops-central](https://console.cloud.google.com/sql/instances/ops-central/backups?project=gitlab-ops) and find the backup marked **Pre-upgrade backup, POSTGRES_16 to POSTGRES_17** (or the post-migration snapshot taken at the end of Phase 1). - [ ] Click 'Restore'. Select 'Overwrite existing instance' and select the instance you created earlier. Click 'Restore'. - [ ] Do whatever needs to be done to connect the application to the restored instance. - [ ] Remove GitLab from maintenance mode \[[instructions](https://docs.gitlab.com/administration/maintenance_mode/#enable-maintenance-mode)\]. - [ ] If [MR 15169](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15169) was already applied/merged, revert it so Terraform state matches the rolled-back (PG16) instance. - [ ] Validate the revert has been successful. - [ ] Set label ~"change::aborted" `/label ~change::aborted`. ## Monitoring ### Key metrics to observe - **Metric:** ops instance metrics - Location: [ops-gitlab-net: Overview](https://dashboards.gitlab.net/d/ops-gitlab-net-main/ops-gitlab-net-overview) - What changes to this metric should prompt a rollback: Extended Apdex and Error Ratio issues (a brief 5XX burst during the DB repoint and the upgrade is expected; sustained errors are not). - **Metric:** Database connectivity / query latency from `ops.gitlab.net` after the Phase 1 repoint (confirms the app is talking to `ops-central` in us-central1). - Location: [ops-gitlab-net: Overview](https://dashboards.gitlab.net/d/ops-gitlab-net-main/ops-gitlab-net-overview) (database panels) / [nonprod-log.gitlab.net](https://nonprod-log.gitlab.net) - What changes to this metric should prompt a rollback: sustained DB connection errors or elevated latency after the repoint. - **Metric:** CI Runners ops environment - Location: [ci-runners: Overview](https://dashboards.gitlab.net/d/ci-runners-main/ci-runners-overview) - What changes to this metric should prompt a rollback: Extended Apdex and Error Ratio issues. - **Metric:** ops-central instance insights - Location: [GCP Console > SQL > ops-central](https://console.cloud.google.com/sql/instances/ops-central/overview?project=gitlab-ops) - What changes to this metric should prompt a rollback: Any errors with the restore or the upgrade, the instance not returning to `RUNNABLE`, or the HA standby failing to resync. ## Change Reviewer checklist <!--To be filled out by the reviewer.--> ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [x] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ## Change Technician checklist <!--To find out who is on-call, use the `@sre-oncall` handle in slack--> - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] This Change Issue is linked to the appropriate Issue and/or Epic. - [ ] Change has been tested in staging and results noted in a comment on this issue. **N/A** - this is the ops instance. - [ ] A dry-run has been conducted and results noted in a comment on this issue. - [x] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [x] For ~C1 and ~C2 change issues, the change event is added to the GitLab Production calendar. - [x] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process.