Upgrade errors from v18.8.7 to v18.9.3 and v18.10.x
<!--IssueSummary start-->
<details>
<summary>
Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards.
</summary>
- [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=595725)
- [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=595725)
</details>
<!--IssueSummary end-->
# **Title**
- Upgrade errors from v18.8.7 to v18.9.3 and v18.10.x
# Summary
- Upgrading from v18.8.7 to v18.9.3 and v18.10.x results in errors.
- This error occurred on two of the three servers on which the upgrade was performed. Is this a known issue?
# **Versions**
- \[x\] Self-managed (CE v18.8.7)
- \[ \] GitLab.com SaaS
- \[ \] Dedicated
# **Current configration**
```
sudo gitlab-rake gitlab:env:info
System information
System: Ubuntu 22.04
Current User: git
Using RVM: no
Ruby Version: 3.2.8
Gem Version: 3.7.1
Bundler Version:2.7.1
Rake Version: 13.0.6
Redis Version: 7.2.11
Sidekiq Version:7.3.9
Go Version: unknown
GitLab information
Version: 18.8.7
Revision: ****
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 16.11
URL: https://****
HTTP Clone URL: https://****/some-group/some-project.git
SSH Clone URL: git@****:some-group/some-project.git
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.45.5
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Gitaly
- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 18.8.7
- default Git Version: 2.52.GIT
```
# **Status of migration jobs before the upgrade**
- CLI
- No job.
```
$ sudo gitlab-rake gitlab:background_migrations:list
id | table_name | job_class_name | status
---|------------|----------------|-------
$ sudo gitlab-psql -c "SELECT job_class_name, table_name, column_name, job_arguments FROM batched_background_migrations WHERE status NOT IN(3, 6);"
job_class_name | table_name | column_name | job_arguments
----------------+------------+-------------+---------------
(0 rows)
```
- UI
- No job.
{width=768 height=357}
# **Upgrade path where errors occur**
- v18.8.7 -> **18.9.3 ERR**
- v18.8.7 -> 18.9.0 OK -> 18.9.2 OK -> **18.9.3 ERR**
- v18.8.7 -> **18.10.0 ERR**
- v18.8.7 -> **18.10.1 ERR**
# ERROR LOG (log excerpt)
- The same error occurred in all of the above upgrade paths.
```
Caused by:
PG::UndefinedTable: ERROR: relation "label_links_archived" does not exist
LINE 9: INSERT INTO label_links_archived (id, label_id, target_id, t...
^
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:229:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:131:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/backfill_sharding_key_and_clean_label_links_table.rb:72:in `delete_and_archive_label_links_with_no_namespace'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/backfill_sharding_key_and_clean_label_links_table.rb:14:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/batched_migration_job.rb:137:in `block (2 levels) in each_sub_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/batch/metrics.rb:21:in `instrument_operation'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/batched_migration_job.rb:136:in `block in each_sub_batch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:102:in `block (2 levels) in each_batch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:102:in `block in each_batch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:72:in `step'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:72:in `each_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/batched_migration_job.rb:135:in `each_sub_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/backfill_sharding_key_and_clean_label_links_table.rb:10:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:62:in `execute_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:34:in `run_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:156:in `run_migration_while'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:95:in `block in execute_migration'
<internal:kernel>:90:in `tap'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:89:in `execute_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:79:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:154:in `block (2 levels) in finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzers/base.rb:58:in `with_suppressed'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:153:in `block in finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:152:in `finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:245:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20251205040000_execute_batched_background_migrations_affected_by_single_record_bug_backport_1.rb:51:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20251205040000_execute_batched_background_migrations_affected_by_single_record_bug_backport_1.rb:45:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20251205040000_execute_batched_background_migrations_affected_by_single_record_bug_backport_1.rb:45:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:94:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/require_disable_ddl_transaction_for_multiple_locks.rb:40:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:46:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:14:in `with_advisory_lock'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:226:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:182:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:114:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
Running handlers complete
[2026-04-03T13:29:32+09:00] ERROR: Exception handlers complete
Infra Phase failed. 12 resources updated in 34 seconds
[2026-04-03T13:29:32+09:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2026-04-03T13:29:32+09:00] FATAL: ---------------------------------------------------------------------------------------
[2026-04-03T13:29:32+09:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2026-04-03T13:29:32+09:00] FATAL: ---------------------------------------------------------------------------------------
[2026-04-03T13:29:32+09:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: Skipping Topology Service health check due to the cell being disabled
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 67320, pg_backend_pid: 51025
main: == 20251205040000 ExecuteBatchedBackgroundMigrationsAffectedBySingleRecordBugBackport1: migrating
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: == [advisory_lock_connection] object_id: 67320, pg_backend_pid: 51025
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedTable: ERROR: relation "label_links_archived" does not exist
LINE 9: INSERT INTO label_links_archived (id, label_id, target_id, t...
..............
Caused by:
PG::UndefinedTable: ERROR: relation "label_links_archived" does not exist
LINE 9: INSERT INTO label_links_archived (id, label_id, target_id, t...
^
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:229:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:131:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/backfill_sharding_key_and_clean_label_links_table.rb:72:in `delete_and_archive_label_links_with_no_namespace'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/backfill_sharding_key_and_clean_label_links_table.rb:14:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/batched_migration_job.rb:137:in `block (2 levels) in each_sub_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/batch/metrics.rb:21:in `instrument_operation'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/batched_migration_job.rb:136:in `block in each_sub_batch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:102:in `block (2 levels) in each_batch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:102:in `block in each_batch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:72:in `step'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:72:in `each_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/batched_migration_job.rb:135:in `each_sub_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/backfill_sharding_key_and_clean_label_links_table.rb:10:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:62:in `execute_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:34:in `run_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:156:in `run_migration_while'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:95:in `block in execute_migration'
<internal:kernel>:90:in `tap'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:89:in `execute_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:79:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:154:in `block (2 levels) in finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzers/base.rb:58:in `with_suppressed'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:153:in `block in finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:152:in `finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:245:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20251205040000_execute_batched_background_migrations_affected_by_single_record_bug_backport_1.rb:51:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20251205040000_execute_batched_background_migrations_affected_by_single_record_bug_backport_1.rb:45:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20251205040000_execute_batched_background_migrations_affected_by_single_record_bug_backport_1.rb:45:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:94:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/require_disable_ddl_transaction_for_multiple_locks.rb:40:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:46:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:14:in `with_advisory_lock'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:226:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:182:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:114:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===
dpkg: error processing package gitlab-ce (--configure):
installed gitlab-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
gitlab-ce
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
```
[UpgradeError.log](/uploads/c37913d95db6d0b89471b6d8aec6bb71/UpgradeError.log)
Best regards,
Masaru Takahashi
issue