rails_migration[gitlab-rails] Error executing action run from ee-15.1.3 to ee-15.2.0
Summary
Error when update Omnibus from gitlab-ee-15.1.3-ee.0.el7.x86_64 to gitlab-ee-15.2.0-ee.0.el7.x86_64 on CentOS 7.9.2009
What is the current bug behavior?
GitLab isn't working, HTTP 503 returned.
gitlab-ctl status
run: gitaly: (pid 12268) 2125s; run: log: (pid 13736) 1555151s down: gitlab-pages: 2269s, normally up; run: log: (pid 13741) 1555151s down: gitlab-workhorse: 2269s, normally up; run: log: (pid 13740) 1555151s down: logrotate: 2268s, normally up; run: log: (pid 13748) 1555151s run: postgresql: (pid 13735) 1555151s; run: log: (pid 13734) 1555151s down: puma: 2267s, normally up; run: log: (pid 13737) 1555151s run: redis: (pid 13739) 1555151s; run: log: (pid 13733) 1555151s down: registry: 2267s, normally up; run: log: (pid 13738) 1555151s down: sidekiq: 2261s, normally up; run: log: (pid 13745) 1555151s
Relevant logs and/or screenshots
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
[2022-07-22T19:48:35+02:00] FATAL: Stacktrace dumped to chef-stacktrace.out
Results of GitLab application Check
Expand for gitlab-ctl reconfigure complete output
[2022-07-22T19:47:29+02:00] INFO: Started Chef Infra Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
Chef Infra Client, version 17.10.0
Patents: https://www.chef.io/patents
Infra Phase starting
[2022-07-22T19:47:29+02:00] INFO: *** Chef Infra Client 17.10.0 ***
[2022-07-22T19:47:29+02:00] INFO: Platform: x86_64-linux
[2022-07-22T19:47:29+02:00] INFO: Chef-client pid: 25226
[2022-07-22T19:47:34+02:00] INFO: Setting the run_list to ["recipe[gitlab-ee]"] from CLI options
[2022-07-22T19:47:35+02:00] INFO: Run List is [recipe[gitlab-ee]]
[2022-07-22T19:47:35+02:00] INFO: Run List expands to [gitlab-ee]
[2022-07-22T19:47:35+02:00] INFO: Starting Chef Infra Client Run for [omitted]
[2022-07-22T19:47:35+02:00] INFO: Running start handlers
[2022-07-22T19:47:35+02:00] INFO: Start handlers complete.
Resolving cookbooks for run list: ["gitlab-ee"]
[2022-07-22T19:47:36+02:00] INFO: Loading cookbooks [gitlab-ee@0.0.1, package@0.1.0, gitlab@0.0.1, consul@0.1.0, patroni@0.1.0, pgbouncer@0.1.0, spamcheck@0.1.0, runit@5.1.3, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, acme@4.1.5, crond@0.1.0]
Synchronizing cookbooks:
- gitlab-ee (0.0.1)
- package (0.1.0)
- patroni (0.1.0)
- consul (0.1.0)
- pgbouncer (0.1.0)
- gitlab (0.0.1)
- spamcheck (0.1.0)
- runit (5.1.3)
- logrotate (0.1.0)
- postgresql (0.1.0)
- redis (0.1.0)
- praefect (0.1.0)
- monitoring (0.1.0)
- letsencrypt (0.1.0)
- mattermost (0.1.0)
- gitaly (0.1.0)
- nginx (0.1.0)
- registry (0.1.0)
- crond (0.1.0)
- gitlab-kas (0.1.0)
- gitlab-pages (0.1.0)
- acme (4.1.5)
Installing cookbook gem dependencies:
Compiling cookbooks...
Top level ::CompositeIO is deprecated, require 'multipart/post' and use `Multipart::Post::CompositeReadIO` instead!
Top level ::Parts is deprecated, require 'multipart/post' and use `Multipart::Post::Parts` instead!
Loading Cinc Auditor profile files:
Loading Cinc Auditor input files:
Loading Cinc Auditor waiver files:
Recipe: gitlab::default
* directory[/etc/gitlab] action create (up to date)
[2022-07-22T19:47:45+02:00] WARN: Selected systemd because /run/systemd/system/ exists
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/ohai-17.9.0/lib/ohai/plugins/rpm.rb:25: warning: already initialized constant MACROS_MARKER
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/ohai-17.9.0/lib/ohai/plugins/rpm.rb:25: warning: previous definition of MACROS_MARKER was here
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/ohai-17.9.0/lib/ohai/plugins/rpm.rb:27: warning: already initialized constant DO_NOT_SPLIT
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/ohai-17.9.0/lib/ohai/plugins/rpm.rb:27: warning: previous definition of DO_NOT_SPLIT was here
Converging 263 resources
* directory[/etc/gitlab] action create (up to date)
* directory[Create /var/opt/gitlab] action create (up to date)
* directory[Create /var/log/gitlab] action create (up to date)
* directory[/opt/gitlab/embedded/etc] action create (up to date)
* template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
* account[Webserver user and group] action create
* group[Webserver user and group] action create (up to date)
* linux_user[Webserver user and group] action create (up to date)
(up to date)
Recipe: gitlab::users
* directory[/var/opt/gitlab] action create (up to date)
* account[GitLab user and group] action create
* group[GitLab user and group] action create (up to date)
* linux_user[GitLab user and group] action create (up to date)
(up to date)
* template[/var/opt/gitlab/.gitconfig] action create (up to date)
* directory[/var/opt/gitlab/.bundle] action create (up to date)
Recipe: gitlab::gitlab-shell
* storage_directory[/var/opt/gitlab/.ssh] action create
* ruby_block[directory resource: /var/opt/gitlab/.ssh] action run (skipped due to not_if)
(up to date)
* directory[/var/log/gitlab/gitlab-shell/] action create (up to date)
* directory[/var/opt/gitlab/gitlab-shell] action create (up to date)
* templatesymlink[Create a config.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
(up to date)
* link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)
* file[/var/opt/gitlab/.ssh/authorized_keys] action create_if_missing (up to date)
Recipe: gitlab::gitlab-rails
* storage_directory[/var/opt/gitlab/git-data] action create
* ruby_block[directory resource: /var/opt/gitlab/git-data] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/git-data/repositories] action create
* ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run (skipped due to not_if)
(up to date)
Recipe: gitlab::rails_pages_shared_path
* storage_directory[/var/opt/gitlab/gitlab-rails/shared] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/pages] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/pages] action run (skipped due to not_if)
(up to date)
Recipe: gitlab::gitlab-rails
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/artifacts] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/external-diffs] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/external-diffs] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/lfs-objects] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/packages] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/packages] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/dependency_proxy] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/dependency_proxy] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/terraform_state] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/terraform_state] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/ci_secure_files] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/ci_secure_files] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/encrypted_settings] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/encrypted_settings] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/uploads] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/uploads] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-ci/builds] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-ci/builds] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/cache] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/cache] action run (skipped due to not_if)
(up to date)
* storage_directory[/var/opt/gitlab/gitlab-rails/shared/tmp] action create
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/tmp] action run (skipped due to not_if)
(up to date)
* storage_directory[/opt/gitlab/embedded/service/gitlab-rails/public] action create (skipped due to only_if)
* directory[create /var/opt/gitlab/gitlab-rails/etc] action create (up to date)
* directory[create /opt/gitlab/etc/gitlab-rails] action create (up to date)
* directory[create /var/opt/gitlab/gitlab-rails/working] action create (up to date)
* directory[create /var/opt/gitlab/gitlab-rails/tmp] action create (up to date)
* directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] action create (up to date)
* directory[create /var/log/gitlab/gitlab-rails] action create (up to date)
* storage_directory[/opt/backup/gitlab] action create
* ruby_block[directory resource: /opt/backup/gitlab] action run (skipped due to not_if)
(up to date)
* directory[/var/opt/gitlab/gitlab-rails] action create (up to date)
* directory[/var/opt/gitlab/gitlab-ci] action create (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key] action create[2022-07-22T19:47:48+02:00] WARN: only_if block for file[/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key] returned a string, did you mean to run a command?
(up to date)
* template[/opt/gitlab/etc/gitlab-rails-rc] action create (up to date)
* file[/opt/gitlab/embedded/service/gitlab-rails/.secret] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/secret] action delete (up to date)
* templatesymlink[Create a database.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/database.yml to /var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
(up to date)
* templatesymlink[Create a secrets.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml to /var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
(up to date)
* templatesymlink[Create a resque.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/resque.yml to /var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
(up to date)
* templatesymlink[Create a cable.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/cable.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/cable.yml to /var/opt/gitlab/gitlab-rails/etc/cable.yml] action create (up to date)
(up to date)
* templatesymlink[Create a redis.cache.yml and create a symlink to Rails root] action create (skipped due to not_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/config/redis.cache.yml] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/redis.cache.yml] action delete (up to date)
* templatesymlink[Create a redis.queues.yml and create a symlink to Rails root] action create (skipped due to not_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/config/redis.queues.yml] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/redis.queues.yml] action delete (up to date)
* templatesymlink[Create a redis.shared_state.yml and create a symlink to Rails root] action create (skipped due to not_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/config/redis.shared_state.yml] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/redis.shared_state.yml] action delete (up to date)
* templatesymlink[Create a redis.trace_chunks.yml and create a symlink to Rails root] action create (skipped due to not_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/config/redis.trace_chunks.yml] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/redis.trace_chunks.yml] action delete (up to date)
* templatesymlink[Create a redis.rate_limiting.yml and create a symlink to Rails root] action create (skipped due to not_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/config/redis.rate_limiting.yml] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/redis.rate_limiting.yml] action delete (up to date)
* templatesymlink[Create a redis.sessions.yml and create a symlink to Rails root] action create (skipped due to not_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/config/redis.sessions.yml] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/redis.sessions.yml] action delete (up to date)
* templatesymlink[Create a smtp_settings.rb and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb to /var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action create (up to date)
(up to date)
* templatesymlink[Create a gitlab.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml to /var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
(up to date)
* templatesymlink[Create a gitlab_workhorse_secret and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
(up to date)
* templatesymlink[Create a gitlab_shell_secret and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
(up to date)
* templatesymlink[Create a gitlab_incoming_email_secret and create a symlink to Rails root] action create (skipped due to only_if)
* templatesymlink[Create a gitlab_service_desk_email_secret and create a symlink to Rails root] action create (skipped due to only_if)
* templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] action create[2022-07-22T19:47:50+02:00] WARN: only_if block for templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_pages_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] action create (up to date)
(up to date)
* templatesymlink[Create a gitlab_kas_secret and create a symlink to Rails root] action create[2022-07-22T19:47:50+02:00] WARN: only_if block for templatesymlink[Create a gitlab_kas_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_kas_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] action create (up to date)
(up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/config/initializers/relative_url.rb] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
* env_dir[/opt/gitlab/etc/gitlab-rails/env] action create
* directory[/opt/gitlab/etc/gitlab-rails/env] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/HOME] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/LD_PRELOAD] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/PATH] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/ICU_DATA] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/PYTHONPATH] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/TZ] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/SSL_CERT_DIR] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/SSL_CERT_FILE] action create (up to date)
(up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/tmp] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/log] action create (up to date)
* link[/var/log/gitlab/gitlab-rails/sidekiq.log] action delete (skipped due to only_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/db/structure.sql] action create (up to date)
* remote_file[/var/opt/gitlab/gitlab-rails/VERSION] action create (up to date)
* remote_file[/var/opt/gitlab/gitlab-rails/REVISION] action create (up to date)
* version_file[Create version file for Rails] action create
* file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] action create (up to date)
(up to date)
* execute[clear the gitlab-rails cache] action nothing (skipped due to action :nothing)
* file[/var/opt/gitlab/gitlab-rails/config.ru] action delete (up to date)
Recipe: gitlab::selinux
* execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-7.2.0-ssh-keygen.pp] action runDisabled
[2022-07-22T19:47:51+02:00] INFO: execute[Guard resource] ran successfully
(skipped due to not_if)
* execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-10.5.0-ssh-authorized-keys.pp] action runDisabled
[2022-07-22T19:47:51+02:00] INFO: execute[Guard resource] ran successfully
(skipped due to not_if)
* execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp] action runDisabled
[2022-07-22T19:47:51+02:00] INFO: execute[Guard resource] ran successfully
(skipped due to not_if)
* bash[Set proper security context on ssh files for selinux] action nothing (skipped due to action :nothing)
Recipe: gitlab::add_trusted_certs
* directory[/etc/gitlab/trusted-certs] action create (up to date)
* directory[/opt/gitlab/embedded/ssl/certs] action create (up to date)
* file[/opt/gitlab/embedded/ssl/certs/README] action create (up to date)
* ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run (skipped due to only_if)
Recipe: gitlab::default
* service[create a temporary puma service] action nothing (skipped due to action :nothing)
* service[create a temporary sidekiq service] action nothing (skipped due to action :nothing)
* service[create a temporary mailroom service] action nothing (skipped due to action :nothing)
Recipe: package::runit_systemd
* directory[/usr/lib/systemd/system] action create (up to date)
* template[/usr/lib/systemd/system/gitlab-runsvdir.service] action create (up to date)
* file[/etc/systemd/system/default.target.wants/gitlab-runsvdir.service] action delete (up to date)
* file[/etc/systemd/system/basic.target.wants/gitlab-runsvdir.service] action delete (up to date)
* execute[systemctl daemon-reload] action nothing (skipped due to action :nothing)
* execute[systemctl enable gitlab-runsvdir] action nothing (skipped due to action :nothing)
* execute[systemctl start gitlab-runsvdir] action nothing (skipped due to action :nothing)
Recipe: package::sysctl
* execute[reload all sysctl conf] action nothing (skipped due to action :nothing)
Recipe: logrotate::folders_and_configs
* directory[/var/opt/gitlab/logrotate] action create (up to date)
* directory[/var/opt/gitlab/logrotate/logrotate.d] action create (up to date)
* directory[/var/log/gitlab/logrotate] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.conf] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.d/nginx] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.d/puma] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-rails] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-shell] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-pages] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-kas] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.d/gitaly] action create (up to date)
* template[/var/opt/gitlab/logrotate/logrotate.d/mailroom] action create (up to date)
Recipe: logrotate::enable
* service[logrotate] action nothing (skipped due to action :nothing)
* runit_service[logrotate] action enable
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/logrotate] action create (up to date)
* template[/opt/gitlab/sv/logrotate/run] action create (up to date)
* directory[/opt/gitlab/sv/logrotate/log] action create (up to date)
* directory[/opt/gitlab/sv/logrotate/log/main] action create (up to date)
* template[/opt/gitlab/sv/logrotate/log/config] action create (up to date)
* ruby_block[verify_chown_persisted_on_logrotate] action nothing (skipped due to action :nothing)
* link[/var/log/gitlab/logrotate/config] action create (up to date)
* template[/opt/gitlab/sv/logrotate/log/run] action create (up to date)
* directory[/opt/gitlab/sv/logrotate/env] action create (up to date)
* ruby_block[Delete unmanaged env files for logrotate service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/logrotate/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/logrotate/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/logrotate/control] action create (up to date)
* template[/opt/gitlab/sv/logrotate/control/t] action create (up to date)
* link[/opt/gitlab/init/logrotate] action create (up to date)
* file[/opt/gitlab/sv/logrotate/down] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/logrotate] action create (up to date)
* ruby_block[wait for logrotate service socket] action run (skipped due to not_if)
(up to date)
Recipe: redis::enable
* redis_service[redis] action create
* account[user and group for redis] action create
* group[user and group for redis] action create (up to date)
* linux_user[user and group for redis] action create (up to date)
(up to date)
* group[Socket group] action create (up to date)
* directory[/var/opt/gitlab/redis] action create (up to date)
* directory[/var/log/gitlab/redis] action create (up to date)
* template[/var/opt/gitlab/redis/redis.conf] action create (up to date)
* service[redis] action nothing (skipped due to action :nothing)
* runit_service[redis] action enable
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/redis] action create (up to date)
* template[/opt/gitlab/sv/redis/run] action create (up to date)
* directory[/opt/gitlab/sv/redis/log] action create (up to date)
* directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
* template[/opt/gitlab/sv/redis/log/config] action create (up to date)
* ruby_block[verify_chown_persisted_on_redis] action nothing (skipped due to action :nothing)
* link[/var/log/gitlab/redis/config] action create (up to date)
* template[/opt/gitlab/sv/redis/log/run] action create (up to date)
* directory[/opt/gitlab/sv/redis/env] action create (up to date)
* ruby_block[Delete unmanaged env files for redis service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/redis/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/redis/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/redis/control] action create (up to date)
* link[/opt/gitlab/init/redis] action create (up to date)
* file[/opt/gitlab/sv/redis/down] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/redis] action create (up to date)
* ruby_block[wait for redis service socket] action run (skipped due to not_if)
(up to date)
* ruby_block[warn pending redis restart] action run (skipped due to only_if)
(up to date)
* template[/opt/gitlab/etc/gitlab-redis-cli-rc] action create (up to date)
Recipe: gitaly::enable
* directory[/var/opt/gitlab/gitaly] action create (up to date)
* directory[/var/opt/gitlab/gitaly/run] action create (up to date)
* directory[/var/log/gitlab/gitaly] action create (up to date)
* directory[/var/opt/gitlab/gitaly/internal_sockets] action delete (up to date)
* env_dir[/opt/gitlab/etc/gitaly/env] action create
* directory[/opt/gitlab/etc/gitaly/env] action create (up to date)
* file[/opt/gitlab/etc/gitaly/env/HOME] action create (up to date)
* file[/opt/gitlab/etc/gitaly/env/PATH] action create (up to date)
* file[/opt/gitlab/etc/gitaly/env/TZ] action create (up to date)
* file[/opt/gitlab/etc/gitaly/env/PYTHONPATH] action create (up to date)
* file[/opt/gitlab/etc/gitaly/env/ICU_DATA] action create (up to date)
* file[/opt/gitlab/etc/gitaly/env/SSL_CERT_DIR] action create (up to date)
* file[/opt/gitlab/etc/gitaly/env/GITALY_PID_FILE] action create (up to date)
* file[/opt/gitlab/etc/gitaly/env/WRAPPER_JSON_LOGGING] action create (up to date)
(up to date)
* template[Create Gitaly config.toml] action create (up to date)
* service[gitaly] action nothing (skipped due to action :nothing)
* runit_service[gitaly] action enable
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/gitaly] action create (up to date)
* template[/opt/gitlab/sv/gitaly/run] action create (up to date)
* directory[/opt/gitlab/sv/gitaly/log] action create (up to date)
* directory[/opt/gitlab/sv/gitaly/log/main] action create (up to date)
* template[/opt/gitlab/sv/gitaly/log/config] action create (up to date)
* ruby_block[verify_chown_persisted_on_gitaly] action nothing (skipped due to action :nothing)
* link[/var/log/gitlab/gitaly/config] action create (up to date)
* template[/opt/gitlab/sv/gitaly/log/run] action create (up to date)
* directory[/opt/gitlab/sv/gitaly/env] action create (up to date)
* ruby_block[Delete unmanaged env files for gitaly service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/gitaly/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/gitaly/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/gitaly/control] action create (up to date)
* link[/opt/gitlab/init/gitaly] action create (up to date)
* file[/opt/gitlab/sv/gitaly/down] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/gitaly] action create (up to date)
* ruby_block[wait for gitaly service socket] action run (skipped due to not_if)
(up to date)
* version_file[Create version file for Gitaly] action create
* file[/var/opt/gitlab/gitaly/VERSION] action create (up to date)
(up to date)
* version_file[Create Ruby version file for Gitaly] action create
* file[/var/opt/gitlab/gitaly/RUBY_VERSION] action create (up to date)
(up to date)
* consul_service[gitaly] action delete
* file[/var/opt/gitlab/consul/config.d/gitaly-service.json] action delete (up to date)
(up to date)
Recipe: postgresql::bin
* ruby_block[check_postgresql_version] action run (skipped due to not_if)
* ruby_block[check_postgresql_version_is_deprecated] action run (skipped due to not_if)
* ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if)
* template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date)
Recipe: postgresql::user
* account[Postgresql user and group] action create
* group[Postgresql user and group] action create (up to date)
* linux_user[Postgresql user and group] action create (up to date)
(up to date)
* directory[/var/opt/gitlab/postgresql] action create (up to date)
* file[/var/opt/gitlab/postgresql/.profile] action create (up to date)
Recipe: postgresql::sysctl
* gitlab_sysctl[kernel.shmmax] action create
* directory[create /etc/sysctl.d for kernel.shmmax] action create (up to date)
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf kernel.shmmax] action create (up to date)
* link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf] action create (up to date)
* execute[load sysctl conf kernel.shmmax] action nothing (skipped due to action :nothing)
(up to date)
* gitlab_sysctl[kernel.shmall] action create
* directory[create /etc/sysctl.d for kernel.shmall] action create (up to date)
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf kernel.shmall] action create (up to date)
* link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf] action create (up to date)
* execute[load sysctl conf kernel.shmall] action nothing (skipped due to action :nothing)
(up to date)
* gitlab_sysctl[kernel.sem] action create
* directory[create /etc/sysctl.d for kernel.sem] action create (up to date)
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf kernel.sem] action create (up to date)
* link[/etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf] action create (up to date)
* execute[load sysctl conf kernel.sem] action nothing (skipped due to action :nothing)
(up to date)
Recipe: postgresql::enable
* directory[/var/opt/gitlab/postgresql] action create (up to date)
* directory[/var/opt/gitlab/postgresql/data] action create (up to date)
* directory[/var/log/gitlab/postgresql] action create (up to date)
* directory[/var/opt/gitlab/postgresql/data] action create (up to date)
* execute[/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8] action run (skipped due to not_if)
* file[/var/opt/gitlab/postgresql/data/server.crt] action create (up to date)
* file[/var/opt/gitlab/postgresql/data/server.key] action create (up to date)
* postgresql_config[gitlab] action create
* template[/var/opt/gitlab/postgresql/data/postgresql.conf] action create (up to date)
* template[/var/opt/gitlab/postgresql/data/runtime.conf] action create (up to date)
* template[/var/opt/gitlab/postgresql/data/pg_hba.conf] action create (up to date)
* template[/var/opt/gitlab/postgresql/data/pg_ident.conf] action create (up to date)
(up to date)
Recipe: postgresql::standalone
* service[postgresql] action nothing (skipped due to action :nothing)
* runit_service[postgresql] action enable
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/postgresql] action create (up to date)
* template[/opt/gitlab/sv/postgresql/run] action create (up to date)
* directory[/opt/gitlab/sv/postgresql/log] action create (up to date)
* directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date)
* template[/opt/gitlab/sv/postgresql/log/config] action create (up to date)
* ruby_block[verify_chown_persisted_on_postgresql] action nothing (skipped due to action :nothing)
* link[/var/log/gitlab/postgresql/config] action create (up to date)
* template[/opt/gitlab/sv/postgresql/log/run] action create (up to date)
* directory[/opt/gitlab/sv/postgresql/env] action create (up to date)
* ruby_block[Delete unmanaged env files for postgresql service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/postgresql/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/postgresql/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/postgresql/control] action create (up to date)
* template[/opt/gitlab/sv/postgresql/control/t] action create (up to date)
* link[/opt/gitlab/init/postgresql] action create (up to date)
* file[/opt/gitlab/sv/postgresql/down] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/postgresql] action create (up to date)
* ruby_block[wait for postgresql service socket] action run (skipped due to not_if)
* directory[/opt/gitlab/service/postgresql/supervise] action create (up to date)
* directory[/opt/gitlab/service/postgresql/log/supervise] action create (up to date)
* file[/opt/gitlab/service/postgresql/supervise/ok] action touch (skipped due to only_if)
* file[/opt/gitlab/service/postgresql/log/supervise/ok] action touch (skipped due to only_if)
* file[/opt/gitlab/service/postgresql/supervise/status] action touch (skipped due to only_if)
* file[/opt/gitlab/service/postgresql/log/supervise/status] action touch (skipped due to only_if)
* file[/opt/gitlab/service/postgresql/supervise/control] action touch (skipped due to only_if)
* file[/opt/gitlab/service/postgresql/log/supervise/control] action touch (skipped due to only_if)
(up to date)
* database_objects[postgresql] action create
* postgresql_user[gitlab] action create
* execute[create gitlab postgresql user] action run (skipped due to not_if)
(up to date)
* postgresql_user[gitlab_replicator] action create
* execute[create gitlab_replicator postgresql user] action run (skipped due to not_if)
* execute[set options for gitlab_replicator postgresql user] action run (skipped due to not_if)
(up to date)
* postgresql_database[gitlabhq_production] action create
* execute[create database gitlabhq_production] action run (skipped due to not_if)
(up to date)
* postgresql_extension[pg_trgm] action enable
* postgresql_query[enable pg_trgm extension] action run (skipped due to only_if)
(up to date)
* postgresql_extension[btree_gist] action enable
* postgresql_query[enable btree_gist extension] action run (skipped due to only_if)
(up to date)
(up to date)
* version_file[Create version file for PostgreSQL] action create
* file[/var/opt/gitlab/postgresql/VERSION] action create (up to date)
(up to date)
* ruby_block[warn pending postgresql restart] action run (skipped due to only_if)
* execute[reload postgresql] action nothing (skipped due to action :nothing)
* execute[start postgresql] action nothing (skipped due to action :nothing)
Recipe: praefect::disable
* service[praefect] action nothing (skipped due to action :nothing)
* runit_service[praefect] action disable
* ruby_block[disable praefect] action run (skipped due to only_if)
(up to date)
* consul_service[praefect] action delete
* file[/var/opt/gitlab/consul/config.d/praefect-service.json] action delete (up to date)
(up to date)
Recipe: gitlab-kas::disable
* service[gitlab-kas] action nothing (skipped due to action :nothing)
* runit_service[gitlab-kas] action disable
* ruby_block[disable gitlab-kas] action run (skipped due to only_if)
(up to date)
Recipe: gitlab::database_migrations
* ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
* rails_migration[gitlab-rails] action run
* bash[migrate gitlab-rails database] action run
================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'
Resource Declaration:
---------------------
suppressed sensitive resource output
Compiled Resource:
------------------
suppressed sensitive resource output
System Info:
------------
chef_version=17.10.0
platform=centos
platform_version=7.9.2009
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
================================================================================
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
51: rails_migration "gitlab-rails" do
52: rake_task 'gitlab:db:configure'
53: logfile_prefix 'gitlab-rails-db-migrate'
54: helper migration_helper
55:
56: environment env_variables
57: dependent_services dependent_services
58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
59: notifies :run, "ruby_block[check remote PG version]", :immediately
60:
61: only_if { migration_helper.attributes_node['auto_migrate'] }
62: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
rails_migration("gitlab-rails") do
action [:run]
default_guard_interpreter :default
declared_type :rails_migration
cookbook_name "gitlab"
recipe_name "database_migrations"
rake_task "gitlab:db:configure"
logfile_prefix "gitlab-rails-db-migrate"
helper "*sensitive value suppressed*"
environment "*sensitive value suppressed*"
dependent_services []
only_if { #code block }
end
System Info:
------------
chef_version=17.10.0
platform=centos
platform_version=7.9.2009
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
[2022-07-22T19:48:35+02:00] INFO: Running queued delayed notifications before re-raising exception
Running handlers:
[2022-07-22T19:48:35+02:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Running handlers complete
[2022-07-22T19:48:35+02:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 01 minutes 06 seconds
[2022-07-22T19:48:35+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2022-07-22T19:48:35+02:00] FATAL: ---------------------------------------------------------------------------------------
[2022-07-22T19:48:35+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2022-07-22T19:48:35+02:00] FATAL: ---------------------------------------------------------------------------------------
[2022-07-22T19:48:35+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Workaround
- Run
gitlab-psqlto get into the database console. - Then issue this SQL:
UPDATE batched_background_migrations SET max_value = (SELECT max(id) FROM issues) WHERE job_class_name = 'BackfillIssueSearchData';gitlab-ctl restart
Edited by Stan Hu