<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "type::bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary Cannot upgrade self-host omnibus GitLab from 17.4 to 17.5 ### Steps to reproduce direct upgrade omnibus gitlab-ee from `17.4.6` to `17.5.5` using rpm (yum update gitlab-ee-17.5.5) ### Example Project N/A ### What is the current *bug* behavior? main: == 20240917153606 ChangeUpcomingReconciliationsOrganizationIdDefault: migrating main: -- change_column_default("upcoming_reconciliations", "organization_id", nil) main: -> 0.0033s main: == 20240917153606 ChangeUpcomingReconciliationsOrganizationIdDefault: migrated (0.0136s) main: == 20240917192118 DropWorkItemTypesIdDefault: migrating ======================= main: == 20240917192118 DropWorkItemTypesIdDefault: migrated (0.0104s) ==============rake aborted! StandardError: An error has occurred, all later migrations canceled: invalid date /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq-7.2.4/lib/sidekiq/config.rb:163:in `block in redis' /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq-7.2.4/lib/sidekiq/config.rb:160:in `redis' /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq-7.2.4/lib/sidekiq.rb:73:in `redis' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:23:in `block (2 levels) in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_sharding/validator.rb:31:in `allow_unrouted_sidekiq_calls' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:22:in `block in up' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:21:in `each' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:21:in `up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Caused by: Date::Error: invalid date /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq-7.2.4/lib/sidekiq/config.rb:163:in `block in redis' /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq-7.2.4/lib/sidekiq/config.rb:160:in `redis' /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq-7.2.4/lib/sidekiq.rb:73:in `redis' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:23:in `block (2 levels) in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_sharding/validator.rb:31:in `allow_unrouted_sidekiq_calls' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:22:in `block in up' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:21:in `each' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:21:in `up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Caused by: Date::Error: invalid date /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq-7.2.4/lib/sidekiq/config.rb:163:in `block in redis' /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq-7.2.4/lib/sidekiq/config.rb:160:in `redis' /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq-7.2.4/lib/sidekiq.rb:73:in `redis' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:23:in `block (2 levels) in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_sharding/validator.rb:31:in `allow_unrouted_sidekiq_calls' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:22:in `block in up' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:21:in `each' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240920072729_remove_dangling_cron_worker.rb:21:in `up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Tasks: TOP => db:migrate (See full trace by running task with --trace) main: == 20240918055146 RemoveStateChangedAtPipelineIdColumn: migrating ============= main: -- transaction_open?(nil) main: -> 0.0000s main: -- remove_column(:vulnerability_state_transitions, :state_changed_at_pipeline_id, {:if_exists=>true}) main: -> 0.0050s main: == 20240918055146 RemoveStateChangedAtPipelineIdColumn: migrated (0.0186s) ==== ### What is the expected *correct* behavior? Upgrade succeed. ### Relevant logs and/or screenshots <!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise. --> ### Output of checks <!-- If you are reporting a bug on GitLab.com, uncomment below --> <!-- This bug happens on GitLab.com --> <!-- and uncomment below if you have /label privileges --> <!-- /label ~"reproduced on GitLab.com" --> <!-- or follow up with an issue comment of `@gitlab-bot label ~"reproduced on GitLab.com"` if you do not --> #### Results of GitLab environment info <!-- Input any relevant GitLab environment information if needed. --> <details> <summary>Expand for output related to GitLab environment info</summary> <pre> System information System: Proxy: https_proxy: MYPROXY_SERVER http_proxy: MYPROXY_SERVER Current User: git Using RVM: no Ruby Version: 3.2.5 Gem Version: 3.5.17 Bundler Version:2.5.11 Rake Version: 13.0.6 Redis Version: 7.0.15 Sidekiq Version:7.2.4 Go Version: unknown GitLab information Version: 17.5.5-ee Revision: 0340daef062 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 14.11 URL: https://MYGITLABURL HTTP Clone URL: https://MYGITLABURL/some-group/some-project.git SSH Clone URL: git@MYGITLABURL:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 14.39.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 17.5.5 - default Git Version: 2.46.2 </pre> </details> #### Results of GitLab application Check <!-- Input any relevant GitLab application check information if needed. --> <details> <summary>Expand for output related to the GitLab application check</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`) (we will only investigate if the tests are passing) </pre> </details> ### Possible fixes <!-- If you can, link to the line of code that might be responsible for the problem. --> <!-- If you don't have /label privileges, follow up with an issue comment of `@gitlab-bot label ~"type::bug"` -->