After upgrading the docker image tag, pulling the new image and restarting the containers, the database migrations fail with the following error: PG::UndefinedTable: ERROR: relation "index_deployments_on_id_and_status_and_created_at" does not exist On professional_teal_sparrow, the post_deploy_db_migration [job](https://console.gitlab-dedicated.com/tenants/8/amp_jobs/41531/logs) database migration failed with the error: `2026-05-26 06:08:13.313:PG::UndefinedTable: ERROR: relation "bigint_idx_89fb13150c9da167f173" does not exist` @ktchernov was able to resolve the issue by accessing the rails console and adding the missing relation back. slack: <details> ``` failed again: 2026-05-26 06:08:13.305:main: -> 0.0000s 2026-05-26 06:08:13.305:main: -- quote_table_name("index_deployments_on_id_and_status_and_created_at") 2026-05-26 06:08:13.305:main: -> 0.0000s 2026-05-26 06:08:13.305:main: -- execute("ALTER INDEX \"bigint_idx_89fb13150c9da167f173\" RENAME TO \"index_deployments_on_id_and_status_and_created_at\"") 2026-05-26 06:08:13.312:main: == [advisory_lock_connection] object_id: 58580, pg_backend_pid: 13205 2026-05-26 06:08:13.312:bin/rails aborted! 2026-05-26 06:08:13.312:StandardError: An error has occurred, all later migrations canceled: 2026-05-26 06:08:13.312: 2026-05-26 06:08:13.312:PG::UndefinedTable: ERROR: relation "bigint_idx_89fb13150c9da167f173" does not exist 2026-05-26 06:08:13.312:/srv/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:56:in `exec' 2026-05-26 06:08:13.312:/srv/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:56:in `block (2 levels) in raw_execute' 2026-05-26 06:08:13.312:/srv/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract_adapter.rb:1024:in `block in with_raw_connection' 2026-05-26 06:08:13.312:/srv/gitlab/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.3/lib/active_support/concurrency/null_lock.rb:9:in `synchronize' 2026-05-26 06:08:13.312:/srv/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract_adapter.rb:993:in `with_raw_connection'Vamshidhar Poralla  [11:12 PM] can i get a output of the following sql SELECT indexname FROM pg_indexes WHERE tablename = 'deployments' AND indexname LIKE '%status_and_created_at%' OR indexname LIKE '%89fb13%';Kunal  [11:14 PM] give me a few mins please Kunal  [11:15 PM] @gitlab-dedicated/ems permission to break glass Kunal  [11:18 PM] gitlabhq_production=> gitlabhq_production=> SELECT indexname FROM pg_indexes gitlabhq_production-> WHERE tablename = 'deployments' gitlabhq_production-> AND indexname LIKE '%status_and_created_at%' gitlabhq_production-> OR indexname LIKE '%89fb13%'; indexname ----------------------------------------------------------- index_deployments_on_project_id_and_status_and_created_at index_deployments_on_id_and_status_and_created_at (2 rows) gitlabhq_production=>Vamshidhar Poralla  [11:24 PM] The migration's swap_indexes expects both to exist simultaneously. What it needs to do: Rename index_deployments_on_id_and_status_and_created_at → temp_xxxx Rename bigint_idx_89fb13150c9da167f173 → index_deployments_on_id_and_status_and_created_at But since we have already effectively done step 2 manually, the end state is correct — the right index is in place with the right name. Skipping is now actually valid for this specific index. The cleanest path forward: -- Give the migration what it needs for step 1 to succeed, -- then step 2 will rename "bigint" index which doesn't exist... -- so instead, just rename current index to the bigint name so swap can complete fully: ALTER INDEX index_deployments_on_id_and_status_and_created_at RENAME TO bigint_idx_89fb13150c9da167f173;This restores the exact state the migration expects — both indexes exist with their original names — and lets swap_indexes do both renames cleanly itself. Then re-trigger the migration.Kunal  [11:25 PM] ok, running the following: ALTER INDEX index_deployments_on_id_and_status_and_created_at RENAME TO bigint_idx_89fb13150c9da167f173;Kunal  [11:27 PM] done, Is this what you expected? gitlabhq_production=> ALTER INDEX index_deployments_on_id_and_status_and_created_at gitlabhq_production-> RENAME TO bigint_idx_89fb13150c9da167f173; ALTER INDEX gitlabhq_production=> SELECT indexname FROM pg_indexes WHERE tablename = 'deployments' AND indexname LIKE '%status_and_created_at%' OR indexname LIKE '%89fb13%'; indexname ----------------------------------------------------------- index_deployments_on_project_id_and_status_and_created_at bigint_idx_89fb13150c9da167f173 (2 rows) gitlabhq_production=>(edited) Vamshidhar Poralla  [11:31 PM] Create the missing original index: CREATE INDEX CONCURRENTLY index_deployments_on_id_and_status_and_created_at ON deployments (id, status, created_at);Once done you'll have both: SELECT indexname FROM pg_indexes WHERE tablename = 'deployments' AND indexname IN ( 'index_deployments_on_id_and_status_and_created_at', 'bigint_idx_89fb13150c9da167f173' ); index_deployments_on_id_and_status_and_created_at ✓ bigint_idx_89fb13150c9da167f173 ✓ Which is exactly what swap_indexes expects. Then re-trigger the migration and it should complete cleanly. Kunal  [11:32 PM] gitlabhq_production=> CREATE INDEX CONCURRENTLY index_deployments_on_id_and_status_and_created_at gitlabhq_production-> ON deployments (id, status, created_at); ERROR: canceling statement due to statement timeout gitlabhq_production=>Kunal  [11:32 PM] this timed out, is it safe to retry? Kunal  [11:32 PM] or should we look at something else before that Vamshidhar Poralla  [11:33 PM] we need to set stattement timeout to false SET statement_timeout = 0; CREATE INDEX CONCURRENTLY index_deployments_on_id_and_status_and_created_at ON deployments (id, status, created_at);Vamshidhar Poralla  [11:33 PM] This index creation may take several minutes depending on the table size. Kunal  [11:34 PM] previous timeout was misleading, it did create index": gitlabhq_production=> SET statement_timeout = 0; SET gitlabhq_production=> CREATE INDEX CONCURRENTLY index_deployments_on_id_and_status_and_created_at gitlabhq_production-> ON deployments (id, status, created_at); ERROR: relation "index_deployments_on_id_and_status_and_created_at" already exists gitlabhq_production=> SELECT indexname FROM pg_indexes gitlabhq_production-> WHERE tablename = 'deployments' gitlabhq_production-> AND indexname IN ( gitlabhq_production(> 'index_deployments_on_id_and_status_and_created_at', gitlabhq_production(> 'bigint_idx_89fb13150c9da167f173' gitlabhq_production(> ); indexname --------------------------------------------------- index_deployments_on_id_and_status_and_created_at bigint_idx_89fb13150c9da167f173 (2 rows) gitlabhq_production=>Vamshidhar Poralla  [11:34 PM] Please re-trigger the migrations now Kunal  [11:36 PM] ok, running Kunal  [11:42 PM] that went fine, its now running db:migrate:geo Kunal  [11:42 PM] all done, db migration completed successfully Vamshidhar Poralla  [11:42 PM] Thats cool Vamshidhar Poralla  [11:45 PM] Great! To summarise what happened and why: The customer's DB was missing index_deployments_on_id_and_status_and_created_at — it had been cleaned up ahead of schedule as part of their bigint migration progress The swap_indexes logic needed both the original and bigint-named indexes to exist simultaneously to do its two-step rename The back-and-forth renames were us trying to reconstruct that state without creating the missing index The real fix was simply creating the original index. ``` </details> - Inspections of two other tenants that have not yet been upgraded to 18.11.x show that they have the relation that professional_teal_sparrow was missing. - [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/237974) will be in next week's maintenance and will allow this migration to continue without causing failure. ### Next Steps - [x] Document mitigation - [x] re-enable upgrade to 18.11 op APAC tenants - [ ] after APAC tenants update, re-evaluate whether or not to enable upgrade to 18.11 on AMER-2 tenants ### Checking for missing indexes - `Gitlab.com_except_jh?` - `ActiveRecord::Base.connection.indexes('deployments').each { |i| puts "#{i.name} | columns: #{i.columns.join(', ')} | unique: #{i.unique}" }; nil` - `ActiveRecord::Base.connection.columns('deployments').each { |c| puts "#{c.name} | #{c.type} | #{c.sql_type} | null: #{c.null} | default: #{c.default}" }; nil` ### Mitigation - If the index `index_deployments_on_id_and_status_and_created_at` is missing create it with CREATE INDEX CONCURRENTLY index_deployments_on_id_and_status_and_created_at ON deployments (id, status, created_at); - re-run the migration - If the bigint index `bigint_idx_89fb13150c9da167f173` is missing create it with CREATE INDEX CONCURRENTLY bigint_idx_89fb13150c9da167f173 ON deployments (id_convert_to_bigint, status, created_at); - re-run the migration - If you get any statement_timeout errors. run `SET statement_timeout = 0;` and re-run the commands above #### Example: ``` gitlabhq_production=> SET statement_timeout = 0; SET gitlabhq_production=> CREATE INDEX CONCURRENTLY index_deployments_on_id_and_status_and_created_at gitlabhq_production-> ON deployments (id, status, created_at); ERROR: relation "index_deployments_on_id_and_status_and_created_at" already exists gitlabhq_production=> SELECT indexname FROM pg_indexes gitlabhq_production-> WHERE tablename = 'deployments' gitlabhq_production-> AND indexname IN ( gitlabhq_production(> 'index_deployments_on_id_and_status_and_created_at', gitlabhq_production(> 'bigint_idx_89fb13150c9da167f173' gitlabhq_production(> ); indexname --------------------------------------------------- index_deployments_on_id_and_status_and_created_at bigint_idx_89fb13150c9da167f173 (2 rows) gitlabhq_production=> ```