update-major e2e test jobs failing on master

Summary

Failing job(s):

update-major: https://gitlab.com/gitlab-org/gitlab/-/jobs/4204663721 (See Complete Raw for full logs)

The update path is gitlab/gitlab-ee:15.11.0-ee.0 => registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:34b9fcb00be8a05fa173b1fcf83fc954307d30db (16.0.0-pre)

About this test: https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testomnibusupdatefromprevious-full-image-address-current_version-majorminor-from_edition

The update-minor job also fails with the same stack trace and the update path is also the same: https://gitlab.com/gitlab-org/gitlab/-/jobs/4204663783

Stack trace

      StandardError: An error has occurred, this and all later migrations canceled:
      
      PG::DuplicateSchema: ERROR:  schema "gitlab_partitions_dynamic" already exists
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211202041233_init_schema.rb:7:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:369:in `block in transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:368:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:118:in `configure_database'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:96:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:25:in `load'
      /opt/gitlab/embedded/bin/bundle:25:in `<main>'
      
      Caused by:
      ActiveRecord::StatementInvalid: PG::DuplicateSchema: ERROR:  schema "gitlab_partitions_dynamic" already exists
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211202041233_init_schema.rb:7:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:369:in `block in transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:368:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:118:in `configure_database'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:96:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:25:in `load'
      /opt/gitlab/embedded/bin/bundle:25:in `<main>'
      
      Caused by:
      PG::DuplicateSchema: ERROR:  schema "gitlab_partitions_dynamic" already exists
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211202041233_init_schema.rb:7:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:369:in `block in transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:368:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:118:in `configure_database'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:96:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:25:in `load'
      /opt/gitlab/embedded/bin/bundle:25:in `<main>'
      Tasks: TOP => db:migrate
      (See full trace by running task with --trace)
      STDERR: 
      ---- End output of "bash"  ----
      Ran "bash"  returned 1
      
      Cookbook Trace: (most recent call first)
      ----------------------------------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'

Screenshot / HTML page

Possible fixes

Edited May 01, 2023 by Sanad Liaquat