### Summary Trying to upgrade from 16.7.0 to 16.7.2. When upgrading, the container never reached "healthy" state and failed. Note, I initially tried to upgrade to 16.7.1 but could not find reference to it anywhere and no Docker image exists. ### Steps to reproduce Update docker-compose.yml file, change the image to use from `gitlab/gitlab-ce:16.7.0-ce.0` to `gitlab/gitlab-ce:16.7.2-ce.0` - This problem has been reported with both **CE** and **EE**. ### What is the current *bug* behavior? The Docker container never becomes healthy and a rake migration appears to fail. ### What is the expected *correct* behavior? The rake migrations succeed and the Docker container becomes healthy. ### Relevant logs <details> <summary> Relevant logs </summary> ``` * template[/var/opt/gitlab/postgresql/data/pg_ident.conf] action create (up to date) (up to date) Recipe: postgresql::standalone * service[postgresql] action nothing (skipped due to action :nothing) * runit_service[postgresql] action enable * ruby_block[restart_service] action nothing (skipped due to action :nothing) * ruby_block[restart_log_service] action nothing (skipped due to action :nothing) * ruby_block[reload_log_service] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/sv/postgresql] action create[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql] created directory /opt/gitlab/sv/postgresql - create new directory /opt/gitlab/sv/postgresql[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql] owner changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql] group changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * template[/opt/gitlab/sv/postgresql/run] action create[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/run] created file /opt/gitlab/sv/postgresql/run - create new file /opt/gitlab/sv/postgresql/run[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/run] updated file contents /opt/gitlab/sv/postgresql/run - update content in file /opt/gitlab/sv/postgresql/run from none to dc5689 --- /opt/gitlab/sv/postgresql/run 2024-01-12 02:46:20.779972087 +0000 +++ /opt/gitlab/sv/postgresql/.chef-run20240112-33-j17uzp 2024-01-12 02:46:20.779972087 +0000 @@ -1,4 +1,8 @@ +#!/bin/sh +exec 2>&1 + +exec chpst -P -U gitlab-psql:gitlab-psql -u gitlab-psql:gitlab-psql /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/run] owner changed to 0 [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/run] group changed to 0 [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/run] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * directory[/opt/gitlab/sv/postgresql/log] action create[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/log] created directory /opt/gitlab/sv/postgresql/log - create new directory /opt/gitlab/sv/postgresql/log[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/log] owner changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/log] group changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/log] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * directory[/opt/gitlab/sv/postgresql/log/main] action create[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/log/main] created directory /opt/gitlab/sv/postgresql/log/main - create new directory /opt/gitlab/sv/postgresql/log/main[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/log/main] owner changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/log/main] group changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/log/main] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * template[/opt/gitlab/sv/postgresql/log/config] action create[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/config] created file /opt/gitlab/sv/postgresql/log/config - create new file /opt/gitlab/sv/postgresql/log/config[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/config] updated file contents /opt/gitlab/sv/postgresql/log/config - update content in file /opt/gitlab/sv/postgresql/log/config from none to 623c00 --- /opt/gitlab/sv/postgresql/log/config 2024-01-12 02:46:20.801972449 +0000 +++ /opt/gitlab/sv/postgresql/log/.chef-config20240112-33-e1rlok 2024-01-12 02:46:20.801972449 +0000 @@ -1,6 +1,12 @@ +s209715200 +n30 +t86400 +!gzip + +[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/config] owner changed to 0 [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/config] group changed to 0 [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/config] mode changed to 644 - change mode from '' to '0644' - change owner from '' to 'root' - change group from '' to 'root' [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/config] sending create action to ruby_block[verify_chown_persisted_on_postgresql] (immediate) * ruby_block[verify_chown_persisted_on_postgresql] action create[2024-01-12T02:46:20+00:00] INFO: ruby_block[verify_chown_persisted_on_postgresql] called - execute the ruby block verify_chown_persisted_on_postgresql * ruby_block[verify_chown_persisted_on_postgresql] action nothing (skipped due to action :nothing) * link[/var/log/gitlab/postgresql/config] action create (up to date) * template[/opt/gitlab/sv/postgresql/log/run] action create[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/run] created file /opt/gitlab/sv/postgresql/log/run - create new file /opt/gitlab/sv/postgresql/log/run[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/run] updated file contents /opt/gitlab/sv/postgresql/log/run - update content in file /opt/gitlab/sv/postgresql/log/run from none to 9c7bfd --- /opt/gitlab/sv/postgresql/log/run 2024-01-12 02:46:20.819972745 +0000 +++ /opt/gitlab/sv/postgresql/log/.chef-run20240112-33-5glr20 2024-01-12 02:46:20.819972745 +0000 @@ -1,5 +1,10 @@ +#!/bin/sh +exec chpst -P \ + -U root:root \ + -u root:root \ + svlogd -tt /var/log/gitlab/postgresql[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/run] owner changed to 0 [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/run] group changed to 0 [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/run] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * directory[/opt/gitlab/sv/postgresql/env] action create[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/env] created directory /opt/gitlab/sv/postgresql/env - create new directory /opt/gitlab/sv/postgresql/env[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/env] owner changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/env] group changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/env] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * ruby_block[Delete unmanaged env files for postgresql service] action run (skipped due to only_if) * template[/opt/gitlab/sv/postgresql/check] action create (skipped due to only_if) * template[/opt/gitlab/sv/postgresql/finish] action create (skipped due to only_if) * directory[/opt/gitlab/sv/postgresql/control] action create[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/control] created directory /opt/gitlab/sv/postgresql/control - create new directory /opt/gitlab/sv/postgresql/control[2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/control] owner changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/control] group changed to 0 [2024-01-12T02:46:20+00:00] INFO: directory[/opt/gitlab/sv/postgresql/control] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * template[/opt/gitlab/sv/postgresql/control/t] action create[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/control/t] created file /opt/gitlab/sv/postgresql/control/t - create new file /opt/gitlab/sv/postgresql/control/t[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/control/t] updated file contents /opt/gitlab/sv/postgresql/control/t - update content in file /opt/gitlab/sv/postgresql/control/t from none to 05ae12 --- /opt/gitlab/sv/postgresql/control/t 2024-01-12 02:46:20.841973107 +0000 +++ /opt/gitlab/sv/postgresql/control/.chef-t20240112-33-wo6wl7 2024-01-12 02:46:20.841973107 +0000 @@ -1,3 +1,6 @@ +#!/bin/sh +echo "received TERM from runit, sending INT instead to force quit connections" +/opt/gitlab/embedded/bin/sv interrupt postgresql[2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/control/t] owner changed to 0 [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/control/t] group changed to 0 [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/control/t] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * link[/opt/gitlab/init/postgresql] action create[2024-01-12T02:46:20+00:00] INFO: link[/opt/gitlab/init/postgresql] created - create symlink at /opt/gitlab/init/postgresql to /opt/gitlab/embedded/bin/sv * file[/opt/gitlab/sv/postgresql/down] action nothing (skipped due to action :nothing) [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/run] sending run action to ruby_block[restart_service] (delayed) * ruby_block[restart_service] action run (skipped due to only_if) [2024-01-12T02:46:20+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/config] sending create action to ruby_block[reload_log_service] (delayed) * ruby_block[reload_log_service] action create * ruby_block[restart_service] action nothing (skipped due to action :nothing) * ruby_block[restart_log_service] action nothing (skipped due to action :nothing) * ruby_block[reload_log_service] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/sv/postgresql] action create (up to date) * template[/opt/gitlab/sv/postgresql/run] action create (up to date) * directory[/opt/gitlab/sv/postgresql/log] action create (up to date) * directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date) * template[/opt/gitlab/sv/postgresql/log/config] action create (up to date) * ruby_block[verify_chown_persisted_on_postgresql] action nothing (skipped due to action :nothing) * link[/var/log/gitlab/postgresql/config] action create (up to date) * template[/opt/gitlab/sv/postgresql/log/run] action create (up to date) * directory[/opt/gitlab/sv/postgresql/env] action create (up to date) * ruby_block[Delete unmanaged env files for postgresql service] action run (skipped due to only_if) * template[/opt/gitlab/sv/postgresql/check] action create (skipped due to only_if) * template[/opt/gitlab/sv/postgresql/finish] action create (skipped due to only_if) * directory[/opt/gitlab/sv/postgresql/control] action create (up to date) * template[/opt/gitlab/sv/postgresql/control/t] action create (up to date) * link[/opt/gitlab/init/postgresql] action create (up to date) * file[/opt/gitlab/sv/postgresql/down] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/service] action create (up to date) * link[/opt/gitlab/service/postgresql] action create[2024-01-12T02:46:20+00:00] INFO: link[/opt/gitlab/service/postgresql] created - create symlink at /opt/gitlab/service/postgresql to /opt/gitlab/sv/postgresql * ruby_block[wait for postgresql service socket] action run[2024-01-12T02:46:23+00:00] INFO: ruby_block[wait for postgresql service socket] called - execute the ruby block wait for postgresql service socket * directory[/opt/gitlab/service/postgresql/supervise] action create[2024-01-12T02:46:23+00:00] INFO: directory[/opt/gitlab/service/postgresql/supervise] mode changed to 755 - change mode from '0700' to '0755' * directory[/opt/gitlab/service/postgresql/log/supervise] action create[2024-01-12T02:46:23+00:00] INFO: directory[/opt/gitlab/service/postgresql/log/supervise] mode changed to 755 - change mode from '0700' to '0755' * file[/opt/gitlab/service/postgresql/supervise/ok] action touch[2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/ok] created file /opt/gitlab/service/postgresql/supervise/ok - create new file /opt/gitlab/service/postgresql/supervise/ok[2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/ok] owner changed to 996 [2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/ok] group changed to 996 - change owner from '' to 'gitlab-psql' - change group from '' to 'gitlab-psql'[2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/ok] updated atime and mtime to 2024-01-12 02:46:23 +0000 - update utime on file /opt/gitlab/service/postgresql/supervise/ok * file[/opt/gitlab/service/postgresql/log/supervise/ok] action touch[2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/ok] created file /opt/gitlab/service/postgresql/log/supervise/ok - create new file /opt/gitlab/service/postgresql/log/supervise/ok[2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/ok] owner changed to 996 [2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/ok] group changed to 996 - change owner from '' to 'gitlab-psql' - change group from '' to 'gitlab-psql'[2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/ok] updated atime and mtime to 2024-01-12 02:46:23 +0000 - update utime on file /opt/gitlab/service/postgresql/log/supervise/ok * file[/opt/gitlab/service/postgresql/supervise/status] action touch[2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/status] owner changed to 996 [2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/status] group changed to 996 - change owner from 'root' to 'gitlab-psql' - change group from 'root' to 'gitlab-psql'[2024-01-12T02:46:23+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/status] updated atime and mtime to 2024-01-12 02:46:23 +0000 - update utime on file /opt/gitlab/service/postgresql/supervise/status * file[/opt/gitlab/service/postgresql/log/supervise/status] action touch[2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] owner changed to 996 [2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] group changed to 996 - change owner from 'root' to 'gitlab-psql' - change group from 'root' to 'gitlab-psql'[2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] updated atime and mtime to 2024-01-12 02:46:24 +0000 - update utime on file /opt/gitlab/service/postgresql/log/supervise/status * file[/opt/gitlab/service/postgresql/supervise/control] action touch[2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/control] created file /opt/gitlab/service/postgresql/supervise/control - create new file /opt/gitlab/service/postgresql/supervise/control[2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/control] owner changed to 996 [2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/control] group changed to 996 - change owner from '' to 'gitlab-psql' - change group from '' to 'gitlab-psql'[2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/control] updated atime and mtime to 2024-01-12 02:46:24 +0000 - update utime on file /opt/gitlab/service/postgresql/supervise/control * file[/opt/gitlab/service/postgresql/log/supervise/control] action touch[2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/control] created file /opt/gitlab/service/postgresql/log/supervise/control - create new file /opt/gitlab/service/postgresql/log/supervise/control[2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/control] owner changed to 996 [2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/control] group changed to 996 - change owner from '' to 'gitlab-psql' - change group from '' to 'gitlab-psql'[2024-01-12T02:46:24+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/control] updated atime and mtime to 2024-01-12 02:46:24 +0000 - update utime on file /opt/gitlab/service/postgresql/log/supervise/control * file[/var/log/gitlab/postgresql/current] action touch (skipped due to only_if) [2024-01-12T02:46:25+00:00] INFO: ruby_block[reload_log_service] called - execute the ruby block reload_log_service [2024-01-12T02:46:25+00:00] INFO: template[/opt/gitlab/sv/postgresql/log/run] sending run action to ruby_block[restart_log_service] (delayed) * ruby_block[restart_log_service] action run * ruby_block[restart_service] action nothing (skipped due to action :nothing) * ruby_block[restart_log_service] action nothing (skipped due to action :nothing) * ruby_block[reload_log_service] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/sv/postgresql] action create (up to date) * template[/opt/gitlab/sv/postgresql/run] action create (up to date) * directory[/opt/gitlab/sv/postgresql/log] action create (up to date) * directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date) * template[/opt/gitlab/sv/postgresql/log/config] action create (up to date) * ruby_block[verify_chown_persisted_on_postgresql] action nothing (skipped due to action :nothing) * link[/var/log/gitlab/postgresql/config] action create (up to date) * template[/opt/gitlab/sv/postgresql/log/run] action create (up to date) * directory[/opt/gitlab/sv/postgresql/env] action create (up to date) * ruby_block[Delete unmanaged env files for postgresql service] action run (skipped due to only_if) * template[/opt/gitlab/sv/postgresql/check] action create (skipped due to only_if) * template[/opt/gitlab/sv/postgresql/finish] action create (skipped due to only_if) * directory[/opt/gitlab/sv/postgresql/control] action create (up to date) * template[/opt/gitlab/sv/postgresql/control/t] action create (up to date) * link[/opt/gitlab/init/postgresql] action create (up to date) * file[/opt/gitlab/sv/postgresql/down] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/service] action create (up to date) * link[/opt/gitlab/service/postgresql] action create (up to date) * ruby_block[wait for postgresql service socket] action run (skipped due to not_if) * directory[/opt/gitlab/service/postgresql/supervise] action create (up to date) * directory[/opt/gitlab/service/postgresql/log/supervise] action create (up to date) * file[/opt/gitlab/service/postgresql/supervise/ok] action touch (skipped due to only_if) * file[/opt/gitlab/service/postgresql/log/supervise/ok] action touch (skipped due to only_if) * file[/opt/gitlab/service/postgresql/supervise/status] action touch (skipped due to only_if) * file[/opt/gitlab/service/postgresql/log/supervise/status] action touch[2024-01-12T02:46:25+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] owner changed to 996 [2024-01-12T02:46:25+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] group changed to 996 - change owner from 'root' to 'gitlab-psql' - change group from 'root' to 'gitlab-psql'[2024-01-12T02:46:25+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] updated atime and mtime to 2024-01-12 02:46:25 +0000 - update utime on file /opt/gitlab/service/postgresql/log/supervise/status * file[/opt/gitlab/service/postgresql/supervise/control] action touch (skipped due to only_if) * file[/opt/gitlab/service/postgresql/log/supervise/control] action touch (skipped due to only_if) * file[/var/log/gitlab/postgresql/current] action touch (skipped due to only_if) [2024-01-12T02:46:25+00:00] INFO: ruby_block[restart_log_service] called - execute the ruby block restart_log_service * directory[/opt/gitlab/service] action create (up to date) * link[/opt/gitlab/service/postgresql] action create (up to date) * ruby_block[wait for postgresql service socket] action run (skipped due to not_if) * directory[/opt/gitlab/service/postgresql/supervise] action create (up to date) * directory[/opt/gitlab/service/postgresql/log/supervise] action create (up to date) * file[/opt/gitlab/service/postgresql/supervise/ok] action touch (skipped due to only_if) * file[/opt/gitlab/service/postgresql/log/supervise/ok] action touch (skipped due to only_if) * file[/opt/gitlab/service/postgresql/supervise/status] action touch (skipped due to only_if) * file[/opt/gitlab/service/postgresql/log/supervise/status] action touch[2024-01-12T02:46:25+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] owner changed to 996 [2024-01-12T02:46:25+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] group changed to 996 - change owner from 'root' to 'gitlab-psql' - change group from 'root' to 'gitlab-psql'[2024-01-12T02:46:25+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] updated atime and mtime to 2024-01-12 02:46:25 +0000 - update utime on file /opt/gitlab/service/postgresql/log/supervise/status * file[/opt/gitlab/service/postgresql/supervise/control] action touch (skipped due to only_if) * file[/opt/gitlab/service/postgresql/log/supervise/control] action touch (skipped due to only_if) * file[/var/log/gitlab/postgresql/current] action touch (skipped due to only_if) * database_objects[postgresql] action create * postgresql_user[gitlab] action create * execute[create gitlab postgresql user] action run (skipped due to not_if) (up to date) * postgresql_user[gitlab_replicator] action create * execute[create gitlab_replicator postgresql user] action run (skipped due to not_if) * execute[set options for gitlab_replicator postgresql user] action run (skipped due to not_if) (up to date) * postgresql_database[gitlabhq_production] action create * execute[create database gitlabhq_production] action run (skipped due to not_if) (up to date) * postgresql_extension[pg_trgm] action enable * postgresql_query[enable pg_trgm extension] action run (skipped due to only_if) (up to date) * postgresql_extension[btree_gist] action enable * postgresql_query[enable btree_gist extension] action run (skipped due to only_if) (up to date) * postgresql_database[gitlabhq_production] action create * execute[create database gitlabhq_production] action run (skipped due to not_if) (up to date) * postgresql_extension[pg_trgm] action enable * postgresql_query[enable pg_trgm extension] action run (skipped due to only_if) (up to date) * postgresql_extension[btree_gist] action enable * postgresql_query[enable btree_gist extension] action run (skipped due to only_if) (up to date) (up to date) * version_file[Create version file for PostgreSQL] action create * file[/var/opt/gitlab/postgresql/VERSION] action create (up to date) (up to date) * ruby_block[warn pending postgresql restart] action run (skipped due to only_if) * execute[reload postgresql] action nothing (skipped due to action :nothing) * execute[start postgresql] action nothing (skipped due to action :nothing) Recipe: praefect::disable * service[praefect] action nothing (skipped due to action :nothing) * runit_service[praefect] action disable * ruby_block[disable praefect] action run (skipped due to only_if) (up to date) * consul_service[praefect] action delete * file[/var/opt/gitlab/consul/config.d/praefect-service.json] action delete (up to date) (up to date) Recipe: gitlab-kas::enable * directory[/var/opt/gitlab/gitlab-kas] action create (up to date) * directory[/opt/gitlab/etc/gitlab-kas] action create[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas] created directory /opt/gitlab/etc/gitlab-kas - create new directory /opt/gitlab/etc/gitlab-kas[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas] owner changed to 998 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas] mode changed to 700 - change mode from '' to '0700' - change owner from '' to 'git' * directory[/var/log/gitlab/gitlab-kas] action create (up to date) * version_file[Create version file for Gitlab KAS] action create * file[/var/opt/gitlab/gitlab-kas/VERSION] action create[2024-01-12T02:46:26+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/VERSION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-kas/VERSION.chef-20240112024626.170974 [2024-01-12T02:46:26+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/VERSION] updated file contents /var/opt/gitlab/gitlab-kas/VERSION - update content in file /var/opt/gitlab/gitlab-kas/VERSION from d9d98f to 336bfc --- /var/opt/gitlab/gitlab-kas/VERSION 2024-01-12 01:48:38.140954784 +0000 +++ /var/opt/gitlab/gitlab-kas/.chef-VERSION20240112-33-emibf6 2024-01-12 02:46:26.169060736 +0000 @@ -1 +1 @@ -kas version v16.8.0-rc1, commit: b8f8487a, built: 20231219.120535 +kas version v16.8.0-rc1, commit: b8f8487a, built: 20240111.010649 * file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] action create (up to date) * file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] action create (up to date) * file[/var/opt/gitlab/gitlab-kas/redis_password_file] action create (skipped due to only_if) * file[/var/opt/gitlab/gitlab-kas/redis_sentinels_password_file] action create (skipped due to only_if) * template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] action create (up to date) * env_dir[/opt/gitlab/etc/gitlab-kas/env] action create * directory[/opt/gitlab/etc/gitlab-kas/env] action create[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas/env] created directory /opt/gitlab/etc/gitlab-kas/env - create new directory /opt/gitlab/etc/gitlab-kas/env * file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] action create[2024-01-12T02:46:26+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] created file /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR - create new file /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR[2024-01-12T02:46:26+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR - update content in file /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR from none to 4f45cf - suppressed sensitive resource * file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] action create[2024-01-12T02:46:26+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] created file /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL - create new file /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL[2024-01-12T02:46:26+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] updated file contents /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL - update content in file /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL from none to 45b898 - suppressed sensitive resource * service[gitlab-kas] action nothing (skipped due to action :nothing) * runit_service[gitlab-kas] action enable * ruby_block[restart_service] action nothing (skipped due to action :nothing) * ruby_block[restart_log_service] action nothing (skipped due to action :nothing) * ruby_block[reload_log_service] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/sv/gitlab-kas] action create[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] created directory /opt/gitlab/sv/gitlab-kas - create new directory /opt/gitlab/sv/gitlab-kas[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] owner changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] group changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * template[/opt/gitlab/sv/gitlab-kas/run] action create[2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] created file /opt/gitlab/sv/gitlab-kas/run - create new file /opt/gitlab/sv/gitlab-kas/run[2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] updated file contents /opt/gitlab/sv/gitlab-kas/run - update content in file /opt/gitlab/sv/gitlab-kas/run from none to ac037b --- /opt/gitlab/sv/gitlab-kas/run 2024-01-12 02:46:26.270062397 +0000 +++ /opt/gitlab/sv/gitlab-kas/.chef-run20240112-33-bn8whn 2024-01-12 02:46:26.270062397 +0000 @@ -1,11 +1,22 @@ +#!/bin/bash + +# Let runit capture all script error messages +exec 2>&1 + + + +exec chpst -e /opt/gitlab/etc/gitlab-kas/env -P \ + -u git:git \ + -U git:git \ + /opt/gitlab/embedded/bin/gitlab-kas --configuration-file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml[2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] owner changed to 0 [2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] group changed to 0 [2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * directory[/opt/gitlab/sv/gitlab-kas/log] action create[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] created directory /opt/gitlab/sv/gitlab-kas/log - create new directory /opt/gitlab/sv/gitlab-kas/log[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] owner changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] group changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] created directory /opt/gitlab/sv/gitlab-kas/log/main - create new directory /opt/gitlab/sv/gitlab-kas/log/main[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] owner changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] group changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * template[/opt/gitlab/sv/gitlab-kas/log/config] action create[2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] created file /opt/gitlab/sv/gitlab-kas/log/config - create new file /opt/gitlab/sv/gitlab-kas/log/config[2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] updated file contents /opt/gitlab/sv/gitlab-kas/log/config - update content in file /opt/gitlab/sv/gitlab-kas/log/config from none to 623c00 --- /opt/gitlab/sv/gitlab-kas/log/config 2024-01-12 02:46:26.291062743 +0000 +++ /opt/gitlab/sv/gitlab-kas/log/.chef-config20240112-33-obk8p4 2024-01-12 02:46:26.291062743 +0000 @@ -1,6 +1,12 @@ +s209715200 +n30 +t86400 +!gzip + +[2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] owner changed to 0 [2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] group changed to 0 [2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] mode changed to 644 - change mode from '' to '0644' - change owner from '' to 'root' - change group from '' to 'root' [2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] sending create action to ruby_block[verify_chown_persisted_on_gitlab-kas] (immediate) * ruby_block[verify_chown_persisted_on_gitlab-kas] action create[2024-01-12T02:46:26+00:00] INFO: ruby_block[verify_chown_persisted_on_gitlab-kas] called - execute the ruby block verify_chown_persisted_on_gitlab-kas * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing) * link[/var/log/gitlab/gitlab-kas/config] action create (up to date) * template[/opt/gitlab/sv/gitlab-kas/log/run] action create[2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] created file /opt/gitlab/sv/gitlab-kas/log/run - create new file /opt/gitlab/sv/gitlab-kas/log/run[2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] updated file contents /opt/gitlab/sv/gitlab-kas/log/run - update content in file /opt/gitlab/sv/gitlab-kas/log/run from none to 22436c --- /opt/gitlab/sv/gitlab-kas/log/run 2024-01-12 02:46:26.307063006 +0000 +++ /opt/gitlab/sv/gitlab-kas/log/.chef-run20240112-33-t8xpc6 2024-01-12 02:46:26.307063006 +0000 @@ -1,5 +1,10 @@ +#!/bin/sh +exec chpst -P \ + -U root:root \ + -u root:root \ + svlogd -tt /var/log/gitlab/gitlab-kas[2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] owner changed to 0 [2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] group changed to 0 [2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * directory[/opt/gitlab/sv/gitlab-kas/env] action create[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] created directory /opt/gitlab/sv/gitlab-kas/env - create new directory /opt/gitlab/sv/gitlab-kas/env[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] owner changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] group changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if) * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if) * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if) * directory[/opt/gitlab/sv/gitlab-kas/control] action create[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] created directory /opt/gitlab/sv/gitlab-kas/control - create new directory /opt/gitlab/sv/gitlab-kas/control[2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] owner changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] group changed to 0 [2024-01-12T02:46:26+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] mode changed to 755 - change mode from '' to '0755' - change owner from '' to 'root' - change group from '' to 'root' * link[/opt/gitlab/init/gitlab-kas] action create[2024-01-12T02:46:26+00:00] INFO: link[/opt/gitlab/init/gitlab-kas] created - create symlink at /opt/gitlab/init/gitlab-kas to /opt/gitlab/embedded/bin/sv * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing) [2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] sending run action to ruby_block[restart_service] (delayed) * ruby_block[restart_service] action run (skipped due to only_if) [2024-01-12T02:46:26+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] sending create action to ruby_block[reload_log_service] (delayed) * ruby_block[reload_log_service] action create * ruby_block[restart_service] action nothing (skipped due to action :nothing) * ruby_block[restart_log_service] action nothing (skipped due to action :nothing) * ruby_block[reload_log_service] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/sv/gitlab-kas] action create (up to date) * template[/opt/gitlab/sv/gitlab-kas/run] action create (up to date) * directory[/opt/gitlab/sv/gitlab-kas/log] action create (up to date) * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create (up to date) * template[/opt/gitlab/sv/gitlab-kas/log/config] action create (up to date) * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing) * link[/var/log/gitlab/gitlab-kas/config] action create (up to date) * template[/opt/gitlab/sv/gitlab-kas/log/run] action create (up to date) * directory[/opt/gitlab/sv/gitlab-kas/env] action create (up to date) * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if) * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if) * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if) * directory[/opt/gitlab/sv/gitlab-kas/control] action create (up to date) * link[/opt/gitlab/init/gitlab-kas] action create (up to date) * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/service] action create (up to date) * link[/opt/gitlab/service/gitlab-kas] action create[2024-01-12T02:46:26+00:00] INFO: link[/opt/gitlab/service/gitlab-kas] created - create symlink at /opt/gitlab/service/gitlab-kas to /opt/gitlab/sv/gitlab-kas * ruby_block[wait for gitlab-kas service socket] action run[2024-01-12T02:46:30+00:00] INFO: ruby_block[wait for gitlab-kas service socket] called - execute the ruby block wait for gitlab-kas service socket * file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if) [2024-01-12T02:46:31+00:00] INFO: ruby_block[reload_log_service] called - execute the ruby block reload_log_service [2024-01-12T02:46:31+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] sending run action to ruby_block[restart_log_service] (delayed) * ruby_block[restart_log_service] action run * ruby_block[restart_service] action nothing (skipped due to action :nothing) * ruby_block[restart_log_service] action nothing (skipped due to action :nothing) * ruby_block[reload_log_service] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/sv/gitlab-kas] action create (up to date) * template[/opt/gitlab/sv/gitlab-kas/run] action create (up to date) * directory[/opt/gitlab/sv/gitlab-kas/log] action create (up to date) * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create (up to date) * template[/opt/gitlab/sv/gitlab-kas/log/config] action create (up to date) * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing) * link[/var/log/gitlab/gitlab-kas/config] action create (up to date) * template[/opt/gitlab/sv/gitlab-kas/log/run] action create (up to date) * directory[/opt/gitlab/sv/gitlab-kas/env] action create (up to date) * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if) * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if) * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if) * directory[/opt/gitlab/sv/gitlab-kas/control] action create (up to date) * link[/opt/gitlab/init/gitlab-kas] action create (up to date) * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/service] action create (up to date) * link[/opt/gitlab/service/gitlab-kas] action create (up to date) * ruby_block[wait for gitlab-kas service socket] action run (skipped due to not_if) * file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if) [2024-01-12T02:46:31+00:00] INFO: ruby_block[restart_log_service] called - execute the ruby block restart_log_service * directory[/opt/gitlab/service] action create (up to date) * link[/opt/gitlab/service/gitlab-kas] action create (up to date) * ruby_block[wait for gitlab-kas service socket] action run (skipped due to not_if) * file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if) Recipe: gitlab::database_migrations * ruby_block[check remote PG version] action nothing (skipped due to action :nothing) * rails_migration[gitlab-rails] action run[2024-01-12T02:46:31+00:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700. * bash_hide_env[migrate gitlab-rails database] action run [execute] rake aborted! StandardError: An error has occurred, this and all later migrations canceled: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20231219120134_add_token_to_chat_names.rb:9:in `change' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Caused by: ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20231219120134_add_token_to_chat_names.rb:9:in `change' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Caused by: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20231219120134_add_token_to_chat_names.rb:9:in `change' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Tasks: TOP => db:migrate (See full trace by running task with --trace) Running db:migrate rake task main: == [advisory_lock_connection] object_id: 114800, pg_backend_pid: 505 main: == 20231219120134 AddTokenToChatNames: migrating ============================== main: -- add_column(:chat_names, :encrypted_token, :binary) main: == [advisory_lock_connection] object_id: 114800, pg_backend_pid: 505 ================================================================================ Error executing action `run` on resource 'bash_hide_env[migrate gitlab-rails database]' ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ Expected process to exit with [0], but received '1' ---- Begin output of "bash" ---- STDOUT: rake aborted! StandardError: An error has occurred, this and all later migrations canceled: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20231219120134_add_token_to_chat_names.rb:9:in `change' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Caused by: ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20231219120134_add_token_to_chat_names.rb:9:in `change' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Caused by: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20231219120134_add_token_to_chat_names.rb:9:in `change' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Tasks: TOP => db:migrate (See full trace by running task with --trace) Running db:migrate rake task main: == [advisory_lock_connection] object_id: 114800, pg_backend_pid: 505 main: == 20231219120134 AddTokenToChatNames: migrating ============================== main: -- add_column(:chat_names, :encrypted_token, :binary) main: == [advisory_lock_connection] object_id: 114800, pg_backend_pid: 505 STDERR: ---- End output of "bash" ---- Ran "bash" returned 1 Cookbook Trace: (most recent call first) ---------------------------------------- /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file' Resource Declaration: --------------------- # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb 20: bash_hide_env "migrate #{new_resource.name} database" do 21: code <<-EOH 22: set -e 23: log_file="#{logging_settings[:log_directory]}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log" 24: umask 077 25: /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file} 26: STATUS=${PIPESTATUS[0]} 27: chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file} 28: echo $STATUS > #{new_resource.helper.db_migrate_status_file} 29: exit $STATUS 30: EOH 31: 32: environment new_resource.environment if new_resource.property_is_set?(:environment) 33: new_resource.dependent_services.each do |svc| 34: notifies :restart, svc, :immediately 35: end 36: 37: not_if { new_resource.helper.migrated? } 38: sensitive false 39: end 40: end Compiled Resource: ------------------ # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file' bash_hide_env("migrate gitlab-rails database") do action [:run] default_guard_interpreter :default interpreter "bash" declared_type :bash_hide_env cookbook_name "gitlab" recipe_name "database_migrations" code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n chown git:git ${log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-c6697a9cd49e1cfa45cc31b1ecdb90ae-3c9094d88ea\n exit $STATUS\n" environment "*sensitive value suppressed*" sensitive false domain nil user nil not_if { #code block } end System Info: ------------ chef_version=17.10.0 platform=ubuntu platform_version=22.04 ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/cinc-client executable=/opt/gitlab/embedded/bin/cinc-client ================================================================================ Error executing action `run` on resource 'rails_migration[gitlab-rails]' ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of "bash" ---- STDOUT: rake aborted! StandardError: An error has occurred, this and all later migrations canceled: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20231219120134_add_token_to_chat_names.rb:9:in `change' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Caused by: ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20231219120134_add_token_to_chat_names.rb:9:in `change' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Caused by: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20231219120134_add_token_to_chat_names.rb:9:in `change' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in `<main>' Tasks: TOP => db:migrate (See full trace by running task with --trace) Running db:migrate rake task main: == [advisory_lock_connection] object_id: 114800, pg_backend_pid: 505 main: == 20231219120134 AddTokenToChatNames: migrating ============================== main: -- add_column(:chat_names, :encrypted_token, :binary) main: == [advisory_lock_connection] object_id: 114800, pg_backend_pid: 505 STDERR: ---- End output of "bash" ---- Ran "bash" returned 1 Cookbook Trace: (most recent call first) ---------------------------------------- /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file' Resource Declaration: --------------------- # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb 51: rails_migration "gitlab-rails" do 52: rake_task 'gitlab:db:configure' 53: logfile_prefix 'gitlab-rails-db-migrate' 54: helper migration_helper 55: 56: environment env_variables 57: dependent_services dependent_services 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately 59: notifies :run, "ruby_block[check remote PG version]", :immediately 60: 61: only_if { migration_helper.attributes_node['auto_migrate'] } 62: end Compiled Resource: ------------------ # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file' rails_migration("gitlab-rails") do action [:run] default_guard_interpreter :default declared_type :rails_migration cookbook_name "gitlab" recipe_name "database_migrations" rake_task "gitlab:db:configure" logfile_prefix "gitlab-rails-db-migrate" helper "*sensitive value suppressed*" environment "*sensitive value suppressed*" dependent_services [] only_if { #code block } end System Info: ------------ chef_version=17.10.0 platform=ubuntu platform_version=22.04 ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/cinc-client executable=/opt/gitlab/embedded/bin/cinc-client [2024-01-12T02:46:56+00:00] INFO: Running queued delayed notifications before re-raising exception [2024-01-12T02:46:56+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed) Recipe: gitlab::gitlab-rails * execute[clear the gitlab-rails cache] action run[2024-01-12T02:47:20+00:00] INFO: execute[clear the gitlab-rails cache] ran successfully ``` </details> ### Details of package version <details> <summary>Provide the package version installation details</summary> <pre> Docker CE version </pre> </details>