<!-- Please review https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/ for the most recent information on our change plans and execution policies. --> # Production Change ## Change Summary The 2 migrations in [the MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/225419), would be skipped for normal post-deployment migration process, due to hard to acquire exclusive locks on busy `merge_requests` table. We plan to skip both migrations (mark as succeed and never run) for normal post-deployment migration process for `main` DB (`ci` and `sec` will run as they contain no data), and in this CR, we exec them during low-traffic hours. ## Change Details <!-- To automatically add your change to the GitLab Production calendar update the following fields: - Time tracking - Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM) Bot: https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler --> 1. **Services Impacted** - ~"Service::Postgres" 1. **Change Technician** - @nduff 1. **Change Reviewer** - @krasio 1. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-04-12 23:30 UTC 1. **Time tracking** - 20 minutes 1. **Downtime Component** - No downtime required > [!IMPORTANT] > If your change involves scheduled maintenance, add a step to set and > [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) > per our runbooks. This will make sure SLA calculations adjust for the maintenance period. ## Preparation > [!NOTE] > The following checklists must be done in advance, before setting the label ~"change::scheduled" ### Change Reviewer checklist <!-- To be filled out by the reviewer. --> ~C4 ~C3 ~C2 ~C1: - [ ] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [ ] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ### Change Technician checklist - [ ] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [ ] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [ ] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [ ] This Change Issue is linked to the appropriate Issue and/or Epic - [ ] Change has been tested in staging and results noted in a comment on this issue. - [ ] A dry-run has been conducted and results noted in a comment on this issue. - [ ] The change execution window respects the [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl). - [ ] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [ ] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [ ] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### Pre-execution steps > [!NOTE] > The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are > listed below. - [ ] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [ ] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. - [ ] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. ### Change steps - steps to take to execute the change *Estimated Time to Complete (mins)* - Around 20 minutes - [x] Set label ~"change::in-progress" `/label ~change::in-progress` - [x] Ensure your kubernetes context is set to the correct cluster `gprd-gitlab-gke` - [x] Add a PDB to prevent the toolbox pod being evicted during migration ``` kubectl apply -f - <<'EOF' apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: gitlab-migrations-toolbox-pdb namespace: gitlab spec: maxUnavailable: 0 selector: matchLabels: app: toolbox EOF ``` - [x] Exec to migration toolbox pod ``` kubectl -n gitlab exec -it deploy/gitlab-migrations-toolbox -- bash ``` - [x] Open Rails console - [x] Execute the commands below ``` require Rails.root.join('db/post_migrate/20260310071802_sync_bigint_foreign_keys_validation_on_merge_requests_stage_three.rb') SyncBigintForeignKeysValidationOnMergeRequestsStageThree.new.up ``` ``` require Rails.root.join('db/post_migrate/20260310071803_swap_columns_for_merge_requests_bigint_conversion_stage_three.rb') SwapColumnsForMergeRequestsBigintConversionStageThree.new.up ``` ``` require Rails.root.join('db/post_migrate/20260310071804_drop_tmp_bigint_indexes_on_merge_requests_stage_three.rb') DropTmpBigintIndexesOnMergeRequestsStageThree.new.up ``` - [x] Remove PDB used in previous step. ``` kubectl -n gitlab delete pdb gitlab-migrations-toolbox-pdb ``` - [x] Verify `main` DB has same schema as expected <details> <summary>Click to expand</summary> ```sql Table "public.merge_requests" Column | Type | Collation | Nullable | Default ------------------------------------------------+-----------------------------+-----------+----------+-------------------------------------------- id_convert_to_bigint | integer | | not null | 0 target_branch | character varying(510) | | not null | source_branch | character varying(510) | | not null | source_project_id_convert_to_bigint | integer | | | author_id_convert_to_bigint | integer | | | assignee_id_convert_to_bigint | integer | | | title | character varying(510) | | | NULL::character varying created_at | timestamp with time zone | | not null | updated_at | timestamp with time zone | | not null | milestone_id_convert_to_bigint | integer | | | merge_status | character varying(510) | | not null | 'unchecked'::character varying target_project_id_convert_to_bigint | integer | | not null | 0 iid | integer | | | description | text | | | updated_by_id_convert_to_bigint | integer | | | merge_error | text | | | merge_params | text | | | merge_when_pipeline_succeeds | boolean | | not null | false merge_user_id_convert_to_bigint | integer | | | merge_commit_sha | character varying | | | approvals_before_merge | integer | | | rebase_commit_sha | character varying | | | in_progress_merge_commit_sha | character varying | | | lock_version | integer | | | 0 title_html | text | | | description_html | text | | | time_estimate | integer | | | 0 squash | boolean | | not null | false cached_markdown_version | integer | | | last_edited_at | timestamp without time zone | | | last_edited_by_id_convert_to_bigint | integer | | | merge_jid | character varying | | | discussion_locked | boolean | | | latest_merge_request_diff_id_convert_to_bigint | integer | | | allow_maintainer_to_push | boolean | | | true state_id | smallint | | not null | 1 rebase_jid | character varying | | | squash_commit_sha | bytea | | | merge_ref_sha | bytea | | | draft | boolean | | not null | false prepared_at | timestamp with time zone | | | merged_commit_sha | bytea | | | override_requested_changes | boolean | | not null | false head_pipeline_id | bigint | | | imported_from | smallint | | not null | 0 retargeted | boolean | | not null | false id | bigint | | not null | nextval('merge_requests_id_seq'::regclass) source_project_id | bigint | | | author_id | bigint | | | assignee_id | bigint | | | milestone_id | bigint | | | target_project_id | bigint | | not null | updated_by_id | bigint | | | merge_user_id | bigint | | | last_edited_by_id | bigint | | | latest_merge_request_diff_id | bigint | | | Indexes: "merge_requests_pkey" PRIMARY KEY, btree (id) "idx_merge_requests_on_id_and_merge_jid" btree (id, merge_jid) WHERE merge_jid IS NOT NULL AND state_id = 4 "idx_merge_requests_on_merged_state" btree (id) WHERE state_id = 3 "idx_merge_requests_on_source_project_and_branch_state_opened" btree (source_project_id, source_branch) WHERE state_id = 1 "idx_merge_requests_on_unmerged_state_id" btree (id) WHERE state_id <> 3 "idx_mrs_on_target_id_and_created_at_and_state_id" btree (target_project_id, state_id, created_at, id) "index_merge_requests_for_latest_diffs_with_state_merged" btree (latest_merge_request_diff_id, target_project_id) WHERE state_id = 3 "index_merge_requests_on_assignee_id" btree (assignee_id) "index_merge_requests_on_author_id_and_created_at" btree (author_id, created_at) "index_merge_requests_on_author_id_and_id" btree (author_id, id) "index_merge_requests_on_author_id_and_target_project_id" btree (author_id, target_project_id) "index_merge_requests_on_created_at" btree (created_at) "index_merge_requests_on_head_pipeline_id" btree (head_pipeline_id) "index_merge_requests_on_latest_merge_request_diff_id" btree (latest_merge_request_diff_id) "index_merge_requests_on_merge_user_id" btree (merge_user_id) WHERE merge_user_id IS NOT NULL "index_merge_requests_on_milestone_id" btree (milestone_id) "index_merge_requests_on_source_branch" btree (source_branch) "index_merge_requests_on_source_project_id_and_source_branch" btree (source_project_id, source_branch) "index_merge_requests_on_target_branch" btree (target_branch) "index_merge_requests_on_target_project_id_and_created_at_and_id" btree (target_project_id, created_at, id) "index_merge_requests_on_target_project_id_and_iid" UNIQUE, btree (target_project_id, iid) "index_merge_requests_on_target_project_id_and_merged_commit_sha" btree (target_project_id, merged_commit_sha) "index_merge_requests_on_target_project_id_and_source_branch" btree (target_project_id, source_branch) "index_merge_requests_on_target_project_id_and_squash_commit_sha" btree (target_project_id, squash_commit_sha) "index_merge_requests_on_target_project_id_and_target_branch" btree (target_project_id, target_branch) WHERE state_id = 1 AND merge_when_pipeline_succeeds = true "index_merge_requests_on_target_project_id_and_updated_at_and_id" btree (target_project_id, updated_at, id) "index_merge_requests_on_tp_id_and_merge_commit_sha_and_id" btree (target_project_id, merge_commit_sha, id) "index_merge_requests_on_updated_by_id" btree (updated_by_id) WHERE updated_by_id IS NOT NULL "index_on_merge_requests_for_latest_diffs" btree (target_project_id) INCLUDE (id, latest_merge_request_diff_id) Check constraints: "check_970d272570" CHECK (lock_version IS NOT NULL) Foreign-key constraints: "fk_06067f5644" FOREIGN KEY (latest_merge_request_diff_id) REFERENCES merge_request_diffs(id) ON DELETE SET NULL "fk_6149611a04" FOREIGN KEY (assignee_id) REFERENCES users(id) ON DELETE SET NULL "fk_641731faff" FOREIGN KEY (updated_by_id) REFERENCES users(id) ON DELETE SET NULL "fk_6a5165a692" FOREIGN KEY (milestone_id) REFERENCES milestones(id) ON DELETE SET NULL "fk_a6963e8447" FOREIGN KEY (target_project_id) REFERENCES projects(id) ON DELETE CASCADE "fk_ad525e1f87" FOREIGN KEY (merge_user_id) REFERENCES users(id) ON DELETE SET NULL "fk_e719a85f8a" FOREIGN KEY (author_id) REFERENCES users(id) ON DELETE SET NULL "fk_source_project" FOREIGN KEY (source_project_id) REFERENCES projects(id) ON DELETE SET NULL Referenced by: TABLE "environments" CONSTRAINT "fk_01a033a308" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE SET NULL TABLE "merge_request_assignment_events" CONSTRAINT "fk_08f7602bfd" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "scan_result_policy_violations" CONSTRAINT "fk_17ce579abf" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_requests_compliance_violations" CONSTRAINT "fk_290ec1ab02" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "approvals" CONSTRAINT "fk_310d714958" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "agent_activity_events" CONSTRAINT "fk_3af186389b" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE SET NULL TABLE "merge_requests_approval_rules_merge_requests" CONSTRAINT "fk_74e3466397" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_diffs" CONSTRAINT "fk_8483f3258f" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "security_policy_dismissals" CONSTRAINT "fk_bc10da1827" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "duo_workflows_workflows" CONSTRAINT "fk_ed58162ace" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "status_check_responses" CONSTRAINT "fk_f3953d86c6" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "approval_policy_merge_request_bypass_events" CONSTRAINT "fk_f39e177609" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "p_generated_ref_commits" CONSTRAINT "fk_generated_ref_commits_merge_request_id" FOREIGN KEY (project_id, merge_request_iid) REFERENCES merge_requests(target_project_id, iid) ON DELETE CASCADE TABLE "approval_merge_request_rules" CONSTRAINT "fk_rails_004ce82224" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_context_commits" CONSTRAINT "fk_rails_0fe0039f60" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "description_versions" CONSTRAINT "fk_rails_12b144011c" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "resource_state_events" CONSTRAINT "fk_rails_3112bba7dc" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_blocks" CONSTRAINT "fk_rails_364d4bea8b" FOREIGN KEY (blocked_merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_assignees" CONSTRAINT "fk_rails_443443ce6f" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_requests_closing_issues" CONSTRAINT "fk_rails_458eda8667" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_merge_schedules" CONSTRAINT "fk_rails_5294434bc3" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_requests_merge_data" CONSTRAINT "fk_rails_593f9b7924" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "reviews" CONSTRAINT "fk_rails_5ca11d8c31" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_approval_metrics" CONSTRAINT "fk_rails_5cb1ca73f8" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "resource_iteration_events" CONSTRAINT "fk_rails_6830c13ac1" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "resource_state_events" CONSTRAINT "fk_rails_7ddc5f7457" FOREIGN KEY (source_merge_request_id) REFERENCES merge_requests(id) ON DELETE SET NULL TABLE "deployment_merge_requests" CONSTRAINT "fk_rails_86a6d8bf12" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "excluded_merge_requests" CONSTRAINT "fk_rails_8c973feffa" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_cleanup_schedules" CONSTRAINT "fk_rails_92dd0e705c" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "resource_label_events" CONSTRAINT "fk_rails_9851a00031" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "resource_milestone_events" CONSTRAINT "fk_rails_a006df5590" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_user_mentions" CONSTRAINT "fk_rails_aa1b2961b1" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_trains" CONSTRAINT "fk_rails_b374b5225d" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_predictions" CONSTRAINT "fk_rails_b3b78cbcd0" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_reviewers" CONSTRAINT "fk_rails_d9fec24b9d" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_metrics" CONSTRAINT "fk_rails_e6d7c24d1b" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "draft_notes" CONSTRAINT "fk_rails_e753681674" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "merge_request_blocks" CONSTRAINT "fk_rails_e9387863bc" FOREIGN KEY (blocking_merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE TABLE "timelogs" CONSTRAINT "fk_timelogs_merge_requests_merge_request_id" FOREIGN KEY (merge_request_id) REFERENCES merge_requests(id) ON DELETE CASCADE Triggers: merge_requests_loose_fk_trigger AFTER DELETE ON merge_requests REFERENCING OLD TABLE AS old_table FOR EACH STATEMENT EXECUTE FUNCTION insert_into_loose_foreign_keys_deleted_records() trigger_ecc2780007c2 BEFORE INSERT OR UPDATE ON merge_requests FOR EACH ROW EXECUTE FUNCTION trigger_ecc2780007c2() ``` </details> - [x] Set label ~"change::complete" `/label ~change::complete` ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change *Estimated Time to Complete (mins)* - NA - [ ] In case the migration fails after several attempts, then we need to abort this CR. (no rollback is required). - [ ] The main consequences would be, we could not apply changes to `main` DB in migrations. It's very unlikely this would cause any negative impact to the production `main` DB. - [ ] Set label ~"change::aborted" `/label ~change::aborted` ## Monitoring ### Key metrics to observe <!-- * Describe which dashboards and which specific metrics we should be monitoring related to this change using the format below. --> - Metric: Don't think there is anything else to monitor other than watching the output of the execution