# Production Change ## Change Summary Manually execute post-deployment migration `RemoveTmpBigintFkForDeploymentsPhaseTwoRetryTwo` (timestamp `20260811082241`) on the main stage of `gprd` via a Rails console in the Toolbox pod, following the [Running PDM Manually runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/running-pdm-manually.md). This PDM has failed twice through the standard PDM pipeline with `WithLockRetries::AttemptsExhaustedError`. **Contributing cause:** Project deletions cascade to the unindexed `deployments.project_id_convert_to_bigint` FK, forcing a sequential scan that runs up to the ~15s statement timeout while holding a `RowExclusiveLock` on `deployments`. That lock outlives the PDM's max 7s `lock_timeout`, so every `ACCESS EXCLUSIVE` attempt times out. - Corrective Action: [gitlab-com/gl-infra/production-engineering#29538](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29538) - Migration MR: [gitlab-org/gitlab!249509](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/249509) - Migration file: `db/post_migrate/20260811082241_remove_tmp_bigint_fk_for_deployments_phase_two_retry_two.rb` - Runbook: https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/running-pdm-manually.md - Deadline: **2026-08-20** (next monthly release) ## Change Details 1. **Services Impacted** - Patroni (main), `gprd` main stage 1. **Change Technician** - @swainaina @mbobin 1. **Change Reviewer** - @jarv 1. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-08-17 10:30 UTC 1. **Time tracking** - ~15 min worst case before `AttemptsExhaustedError` (~4-6s actual execution per testing) 1. **Downtime Component** - None expected. Toolbox pod protected from eviction via temporary PodDisruptionBudget during execution. > [!NOTE] > To find a **Change Reviewer**, use the [Reviewer Roulette dashboard](https://gitlab-org.gitlab.io/gitlab-roulette/?currentProject=production&order=1) > scoped to the `production` project. Spin the wheel. If roulette selects the current EOC (check > [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW)), > please re-spin to select another reviewer. > [!IMPORTANT] > If your change involves scheduled maintenance, add a step to set and > [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) > per our runbooks. This will make sure SLA calculations adjust for the maintenance period. ## Preparation > [!NOTE] > The following checklists must be done in advance, before setting the label ~"change::scheduled" ### Change Reviewer checklist ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [x] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [x] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [x] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [ ] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### Pre-execution steps > [!NOTE] > The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are > listed below. - [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [x] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. - [x] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [x] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. - [ ] Confirm the PDM has been skipped via ChatOps: `/chatops gitlab run migrations mark 20260811082241` (only after the PDM has been deployed to production). - [ ] Confirm `kubectl` access to the main stage of `gprd` and permission to run `kubectl apply` in `gprd`. - [ ] Check for in-flight/long-running project deletions or `deployments` row locks (`pg_stat_activity` / `pg_locks`) and wait for a quiet window before executing. ### Change steps - steps to take to execute the change *Estimated Time to Complete (mins)* - 15 min worst case (~4-6s actual execution) - [x] Set label ~"change::in-progress" `/label ~change::in-progress` - [x] Connect to `gke_gitlab-production_us-east1_gprd-gitlab-gke`. - [x] Save the following manifest as `/tmp/pdb.yaml` and apply it: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: gitlab-migrations-toolbox-pdb namespace: gitlab spec: maxUnavailable: 0 selector: matchLabels: app: toolbox ``` ```sh kubectl get deploy --selector='app=toolbox' --no-headers --namespace gitlab kubectl apply --dry-run='server' -f /tmp/pdb.yaml kubectl apply -f /tmp/pdb.yaml ``` - [x] Open a Rails console in the Toolbox pod: ```sh kubectl exec -it deploy/gitlab-migrations-toolbox --container toolbox --namespace gitlab -- /bin/bash gitlab-rails console ``` - [x] Within the Rails console, run the post-deployment migration: ```ruby require Rails.root.join('db/post_migrate/20260811082241_remove_tmp_bigint_fk_for_deployments_phase_two_retry_two.rb') RemoveTmpBigintFkForDeploymentsPhaseTwoRetryTwo.new.up ``` - [x] Confirm a successful run: ``` == 20260811082241 RemoveTmpBigintFkForDeploymentsPhaseTwoRetryTwo: migrating == ... == 20260811082241 RemoveTmpBigintFkForDeploymentsPhaseTwoRetryTwo: migrated == ``` > If the migration raises an exception, **do not delete the PDB yet**. Escalate to the authoring team (@mbobin) to investigate. Only delete the PDB once resolved. - [x] Remove the temporary PDB once the migration completes successfully: ```sh kubectl delete -f /tmp/pdb.yaml ``` - [x] Set label ~"change::complete" `/label ~change::complete` ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change *Estimated Time to Complete (mins)* - N/A - [ ] No rollback is possible for this migration. - [ ] Set label ~"change::aborted" `/label ~change::aborted` ## Monitoring ### Key metrics to observe - Metric: Patroni / database saturation - Location: Patroni and database saturation dashboards for `gprd` main - What changes to this metric should prompt a rollback: Significant increase in lock contention or database saturation during migration execution