Upgrade from 14.0.12 to 14.3.6 ce fails because of missing table
Hi !
Summary
Upgrade from 14.0.12 to 14.3.6 ce fails because of missing table
relation "events_id_seq" does not exist
Seems exactly the same issue as #345593 (closed) (Closed without explanation)
Steps to reproduce
With docker-compose.yml:
version: '3.7'
services:
gitlab:
image: gitlab/gitlab-ce:14.3.6-ce.0
container_name: gitlab
hostname: gitlab.me.lan
environment:
- GITLAB_OMNIBUS_CONFIG=external_url 'https://gitlab.me.lan:30002';
volumes:
- /volume1/docker/gitlab/config:/etc/gitlab
- /volume1/docker/gitlab/logs:/var/log/gitlab
- /volume1/docker/gitlab/data:/var/opt/gitlab
ports:
- "30002:30002"
privileged: true
restart: always- Stop the Docker Container
- Change the version from 14.0.12 to 14.3.6
- Restarted (docker-compose up -d)
- Upgrade fails
Example Project
What is the current bug behavior?
The container restart in loops and never finish the database migration/upgrade
What is the expected correct behavior?
The container should start correctly
Relevant logs and/or screenshots
2022-06-27T08:54:20.484953693Z,stdout,Recipe: gitlab::database_migrations[0m
2022-06-27T08:54:20.485384200Z,stdout, * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
2022-06-27T08:54:20.490515854Z,stdout, * rails_migration[gitlab-rails] action run
2022-06-27T08:55:33.261372246Z,stdout, * bash[migrate gitlab-rails database] action run
2022-06-27T08:55:33.262053408Z,stdout, [execute] rake aborted!
"
2022-06-27T08:55:33.262396739Z,stdout," StandardError: An error has occurred, all later migrations canceled:
2022-06-27T08:55:33.262532391Z,stdout,
2022-06-27T08:55:33.262593867Z,stdout, PG::UndefinedTable: ERROR: relation "events_id_seq" does not exist
2022-06-27T08:55:33.262667069Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:67:in `block in swap'
2022-06-27T08:55:33.262759095Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
2022-06-27T08:55:33.262832909Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
2022-06-27T08:55:33.262907210Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
2022-06-27T08:55:33.262981712Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
2022-06-27T08:55:33.263052150Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:422:in `with_lock_retries'
2022-06-27T08:55:33.263122989Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:44:in `swap'
2022-06-27T08:55:33.263193253Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:18:in `up'
2022-06-27T08:55:33.263262229Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
2022-06-27T08:55:33.263631423Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
2022-06-27T08:55:33.263772438Z,stdout, /opt/gitlab/embedded/bin/bundle:23:in `load'
2022-06-27T08:55:33.263842627Z,stdout, /opt/gitlab/embedded/bin/bundle:23:in `<main>'
2022-06-27T08:55:33.263907916Z,stdout,
2022-06-27T08:55:33.263962679Z,stdout, Caused by:
2022-06-27T08:55:33.264037243Z,stdout, ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "events_id_seq" does not exist
2022-06-27T08:55:33.264175058Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:67:in `block in swap'
2022-06-27T08:55:33.264252597Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
2022-06-27T08:55:33.264339573Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
2022-06-27T08:55:33.264410650Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
2022-06-27T08:55:33.264479288Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
2022-06-27T08:55:33.264683117Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:422:in `with_lock_retries'
2022-06-27T08:55:33.264752806Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:44:in `swap'
2022-06-27T08:55:33.264821157Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:18:in `up'
2022-06-27T08:55:33.264890646Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
2022-06-27T08:55:33.264958072Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
2022-06-27T08:55:33.265029323Z,stdout, /opt/gitlab/embedded/bin/bundle:23:in `load'
2022-06-27T08:55:33.265096737Z,stdout, /opt/gitlab/embedded/bin/bundle:23:in `<main>'
2022-06-27T08:55:33.265163313Z,stdout,
2022-06-27T08:55:33.265217664Z,stdout, Caused by:
2022-06-27T08:55:33.265274590Z,stdout, PG::UndefinedTable: ERROR: relation "events_id_seq" does not exist
2022-06-27T08:55:33.265423255Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:67:in `block in swap'
2022-06-27T08:55:33.265492156Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
2022-06-27T08:55:33.265558470Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
2022-06-27T08:55:33.265630171Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
2022-06-27T08:55:33.265699860Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
2022-06-27T08:55:33.265766261Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:422:in `with_lock_retries'
2022-06-27T08:55:33.265832150Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:44:in `swap'
2022-06-27T08:55:33.265900151Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:18:in `up'
2022-06-27T08:55:33.265968827Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
2022-06-27T08:55:33.266039129Z,stdout, /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
2022-06-27T08:55:33.266112067Z,stdout, /opt/gitlab/embedded/bin/bundle:23:in `load'
2022-06-27T08:55:33.266174131Z,stdout, /opt/gitlab/embedded/bin/bundle:23:in `<main>'
2022-06-27T08:55:33.266240107Z,stdout, Tasks: TOP => db:migrate
2022-06-27T08:55:33.266332171Z,stdout, (See full trace by running task with --trace)
2022-06-27T08:55:33.306990532Z,stdout, == 20210622045705 FinalizeEventsBigintConversion: migrating ===================
2022-06-27T08:55:33.307207848Z,stdout, -- transaction_open?()
2022-06-27T08:55:33.307289937Z,stdout, -> 0.0001s
"
2022-06-27T08:55:33.307370551Z,stdout," -- index_exists?(\"events\", :id_convert_to_bigint, {:unique=>true, :name=>\"index_events_on_id_convert_to_bigint\", :algorithm=>:concurrently})
2022-06-27T08:55:33.307507841Z,stdout, -> 0.0256s
2022-06-27T08:55:33.307569280Z,stdout, -- transaction_open?()
2022-06-27T08:55:33.307627631Z,stdout, -> 0.0000s
"
2022-06-27T08:55:33.307687532Z,stdout," -- index_exists?(\"events\", [:project_id, :id_convert_to_bigint], {:name=>\"index_events_on_project_id_and_id_convert_to_bigint\", :algorithm=>:concurrently})
2022-06-27T08:55:33.307763146Z,stdout, -> 0.0201s
2022-06-27T08:55:33.307823209Z,stdout, -- transaction_open?()
2022-06-27T08:55:33.307880835Z,stdout, -> 0.0000s
"
2022-06-27T08:55:33.307941124Z,stdout," -- index_exists?(\"events\", [:project_id, :id_convert_to_bigint], {:order=>{:id_convert_to_bigint=>:desc}, :where=>\"action = 7\", :name=>\"index_events_on_project_id_and_id_bigint_desc_on_merged_action\", :algorithm=>:concurrently})
2022-06-27T08:55:33.308028363Z,stdout, -> 0.0204s
2022-06-27T08:55:33.308088727Z,stdout, -- transaction_open?()
2022-06-27T08:55:33.308155178Z,stdout, -> 0.0000s
2022-06-27T08:55:33.308216254Z,stdout, -- foreign_keys(:push_event_payloads)
2022-06-27T08:55:33.308276705Z,stdout, -> 0.0184s
2022-06-27T08:55:33.308347931Z,stdout, -- execute("SET statement_timeout TO 0")
2022-06-27T08:55:33.308413432Z,stdout, -> 0.0007s
2022-06-27T08:55:33.308476896Z,stdout, -- execute("ALTER TABLE push_event_payloads VALIDATE CONSTRAINT fk_36c74129da_tmp;")
2022-06-27T08:55:33.308547585Z,stdout, -> 0.0013s
2022-06-27T08:55:33.308607036Z,stdout, -- execute("RESET statement_timeout")
2022-06-27T08:55:33.308667674Z,stdout, -> 0.0006s
"
2022-06-27T08:55:33.308727450Z,stdout," -- execute(\"LOCK TABLE events, push_event_payloads IN ACCESS EXCLUSIVE MODE\")
2022-06-27T08:55:33.308798014Z,stdout, -> 0.0010s
2022-06-27T08:55:33.308856065Z,stdout, -- quote_table_name("events")
2022-06-27T08:55:33.308918041Z,stdout, -> 0.0001s
2022-06-27T08:55:33.308975180Z,stdout, -- quote_column_name(:id)
2022-06-27T08:55:33.309034156Z,stdout, -> 0.0000s
2022-06-27T08:55:33.309090182Z,stdout, -- quote_column_name("id_tmp")
2022-06-27T08:55:33.309148308Z,stdout, -> 0.0000s
2022-06-27T08:55:33.309214234Z,stdout, -- execute("ALTER TABLE \"events\" RENAME COLUMN \"id\" TO \"id_tmp\"")
2022-06-27T08:55:33.309284073Z,stdout, -> 0.0012s
2022-06-27T08:55:33.309358749Z,stdout, -- quote_table_name("events")
2022-06-27T08:55:33.309422075Z,stdout, -> 0.0000s
2022-06-27T08:55:33.309478814Z,stdout, -- quote_column_name(:id_convert_to_bigint)
2022-06-27T08:55:33.309536552Z,stdout, -> 0.0000s
2022-06-27T08:55:33.309593341Z,stdout, -- quote_column_name(:id)
2022-06-27T08:55:33.309650242Z,stdout, -> 0.0000s
2022-06-27T08:55:33.309707130Z,stdout, -- execute("ALTER TABLE \"events\" RENAME COLUMN \"id_convert_to_bigint\" TO \"id\"")
2022-06-27T08:55:33.309774744Z,stdout, -> 0.0012s
2022-06-27T08:55:33.309831945Z,stdout, -- quote_table_name("events")
2022-06-27T08:55:33.309890384Z,stdout, -> 0.0000s
2022-06-27T08:55:33.309950685Z,stdout, -- quote_column_name("id_tmp")
2022-06-27T08:55:33.310014886Z,stdout, -> 0.0000s
2022-06-27T08:55:33.310071899Z,stdout, -- quote_column_name(:id_convert_to_bigint)
2022-06-27T08:55:33.310129613Z,stdout, -> 0.0000s
2022-06-27T08:55:33.310186314Z,stdout, -- execute("ALTER TABLE \"events\" RENAME COLUMN \"id_tmp\" TO \"id_convert_to_bigint\"")
2022-06-27T08:55:33.310256590Z,stdout, -> 0.0010s
2022-06-27T08:55:33.310326904Z,stdout, -- quote_table_name("trigger_69523443cc10")
2022-06-27T08:55:33.310387505Z,stdout, -> 0.0001s
2022-06-27T08:55:33.310474719Z,stdout, -- execute("ALTER FUNCTION \"trigger_69523443cc10\" RESET ALL")
2022-06-27T08:55:33.310542295Z,stdout, -> 0.0012s
2022-06-27T08:55:33.310599309Z,stdout, -- execute("ALTER SEQUENCE events_id_seq OWNED BY events.id")
2022-06-27T08:55:33.406130483Z,stdout, [0m
2022-06-27T08:55:33.408545376Z,stdout, ================================================================================[0m
2022-06-27T08:55:33.408630940Z,stdout, [31mError executing action `run` on resource 'bash[migrate gitlab-rails database]'[0m
2022-06-27T08:55:33.408750405Z,stdout, ================================================================================[0m
2022-06-27T08:55:33.408809806Z,stdout,
2022-06-27T08:55:33.408863532Z,stdout,[0m Mixlib::ShellOut::ShellCommandFailed[0m
2022-06-27T08:55:33.408933558Z,stdout, ------------------------------------[0m
"
2022-06-27T08:55:33.408991847Z,stdout," Expected process to exit with [0], but received '1'
2022-06-27T08:55:33.409059873Z,stdout,[0m ---- Begin output of "bash" "/tmp/chef-script20220627-32-ubli1e" ----
2022-06-27T08:55:33.409140787Z,stdout,[0m STDOUT: rake aborted!
"
2022-06-27T08:55:33.409203850Z,stdout,"[0m StandardError: An error has occurred, all later migrations canceled:Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.7.5p203
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.6
Redis Version: 6.0.16
Git Version: 2.33.0.
Sidekiq Version:5.2.9
Go Version: unknown
GitLab information
Version: 14.3.6
Revision: e522118155e
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.7
URL: https://gitlab.me.lan:30002
HTTP Clone URL: https://gitlab.me.lan:30002/some-group/some-project.git
SSH Clone URL: git@gitlabe.me.lan-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 13.21.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)(we will only investigate if the tests are passing)
Complex to do that on a container that stop quickly
root@gitlab:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.21.1 ? ... OK (13.21.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: FAILED - Internal API unreachable
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yesPossible fixes
Thanks a lot!
Edited by Pierre PINON