gitlab-ce:latest image does not start up due unclean shutdown of database and timeout
I can't restart my gitlab-ce image because the postgres database needs longer to start up than the chef timer allows:
2022-03-17_16:50:27.47830 LOG: starting PostgreSQL 12.7 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
2022-03-17_16:50:27.57280 LOG: listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2022-03-17_16:50:28.24907 LOG: database system was interrupted; last known up at 2022-03-17 16:43:10 GMT
2022-03-17_16:50:30.06571 FATAL: the database system is starting up
2022-03-17_16:50:35.08767 FATAL: the database system is starting up
2022-03-17_16:50:40.11947 FATAL: the database system is starting up
2022-03-17_16:50:45.15618 FATAL: the database system is starting up
2022-03-17_16:50:50.19127 FATAL: the database system is starting up
2022-03-17_16:50:55.22508 FATAL: the database system is starting up
2022-03-17_16:51:00.25980 FATAL: the database system is starting up
2022-03-17_16:51:05.29443 FATAL: the database system is starting up
Running handlers:
gitlab-web-1 | There was an error running gitlab-ctl reconfigure:
gitlab-web-1 |
gitlab-web-1 | rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
gitlab-web-1 | ---- Begin output of "bash" "/tmp/chef-script20220317-28-15w87tt" ----
gitlab-web-1 | STDOUT: rake aborted!
gitlab-web-1 | ActiveRecord::ConnectionNotEstablished: FATAL: the database system is starting up
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:111:in `block in read_write'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:95:in `method_missing'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:71:in `block (3 levels) in <top (required)>'
gitlab-web-1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab-web-1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab-web-1 |
gitlab-web-1 | Caused by:
gitlab-web-1 | PG::ConnectionBad: FATAL: the database system is starting up
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:111:in `block in read_write'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:95:in `method_missing'
gitlab-web-1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:71:in `block (3 levels) in <top (required)>'
gitlab-web-1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab-web-1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab-web-1 | Tasks: TOP => gitlab:db:configure
gitlab-web-1 | (See full trace by running task with --trace)
gitlab-web-1 | STDERR:
gitlab-web-1 | ---- End output of "bash" "/tmp/chef-script20220317-28-15w87tt" ----
gitlab-web-1 | Ran "bash" "/tmp/chef-script20220317-28-15w87tt" returned 1
Can I increse the timeout somehow so the database has more time to start? What could I do instead? I tried to downgrade the image but all of them ran in the same problem
Edited by 🤖 GitLab Bot 🤖