unrecognized configuration parameter "wal_keep_segments" after upgrade to version 16
Summary
I have a gitlab instance running in docker. So far it is running on version 15.11.10. Now I wanted to upgrade to version 16 but the container dies after a short time with the error message:
PG::ConnectionBad: could not connect to server: Connection refused
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
After looking at the Postgres log, I saw the following error:
2023-07-02_11:46:31.32233 2023-07-02 11:46:31.322 GMT [384] FATAL: configuration file "/var/opt/gitlab/postgresql/data/runtime.conf" contains errors
2023-07-02_11:46:32.32822 2023-07-02 11:46:32.328 GMT [385] LOG: unrecognized configuration parameter "wal_keep_segments" in file "/var/opt/gitlab/postgresql/data/runtime.conf" line 30
I had a look at the file runtime.conf
, there is a field wal_keep_segments
. But this field was renamed to wal_keep_size
some time ago. (https://postgresqlco.nf/doc/en/param/wal_keep_segments/)
I haven't found a way to change this in the gitlab.rb
file. Here everything concerning the postgres is commented out. I can provide this if needed.
Maybe I am doing something wrong. Thanks already
Steps to reproduce
I have updated a gitlab instance from version 15 to 16. I don't know if it's that easy to recreate. The instance is a bit older, maybe something has disintegrated here over time.
What is the current bug behavior?
The Gitlab container dies
What is the expected correct behavior?
The gitlab container does not die but gitlab starts
Relevant logs
Relevant logs
Running handlers: [2023-07-02T11:33:02+00:00] ERROR: Running exception handlers There was an error running gitlab-ctl reconfigure:rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of "bash" ---- STDOUT: rake aborted! ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in
read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in
write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:inmethod_missing' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in
configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:inblock (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in
load' /opt/gitlab/embedded/bin/bundle:25:in `'Caused by: PG::ConnectionBad: could not connect to server: Connection refused Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in
read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in
write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:inmethod_missing' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in
configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:inblock (3 levels) in <top (required)>' Running handlers complete /opt/gitlab/embedded/bin/bundle:25:in
load' /opt/gitlab/embedded/bin/bundle:25:in `' Tasks: TOP => gitlab:db:configure (See full trace by running task with --trace) STDERR: ---- End output of "bash" ---- Ran "bash" returned 1[2023-07-02T11:33:02+00:00] ERROR: Exception handlers complete Infra Phase failed. 168 resources updated in 01 minutes 42 seconds [2023-07-02T11:33:02+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out [2023-07-02T11:33:02+00:00] FATAL: --------------------------------------------------------------------------------------- [2023-07-02T11:33:02+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT [2023-07-02T11:33:02+00:00] FATAL: --------------------------------------------------------------------------------------- [2023-07-02T11:33:02+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of "bash" ---- STDOUT: rake aborted! ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in
read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in
write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:inmethod_missing' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in
configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:inblock (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in
load' /opt/gitlab/embedded/bin/bundle:25:in `'Caused by: PG::ConnectionBad: could not connect to server: Connection refused Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in
read_write' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in
write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:inmethod_missing' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in
configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:inblock (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in
load' /opt/gitlab/embedded/bin/bundle:25:in `' Tasks: TOP => gitlab:db:configure (See full trace by running task with --trace) STDERR: ---- End output of "bash" ---- Ran "bash" returned 1
Environment details
- Operating System:
linux ubuntu 22 with docker
- Installation Target, remove incorrect values:
- Bare Metal Machine
- Installation Type, remove incorrect values:
- Upgrade from version 15 to 16
- Is there any other software running on the machine: yes, there are running other containers
- Is this a single or multiple node installation? single
- Resources
- CPU: amd 12 core 24 threads cpu
- Memory total: 32 gb
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
nginx['listen_port'] = 80 nginx['listen_https'] = false nginx['proxy_set_headers'] = { "Host" => "$http_host_with_default", "X-Forwarded-For" => "$proxy_add_x_forwarded_for", "X-Forwarded-Proto" => "https", "X-Forwarded-Ssl" => "on", } nginx['real_ip_trusted_addresses'] = ['10.100.0.0/15'] nginx['real_ip_header'] = 'X-Real-IP' nginx['real_ip_recursive'] = 'on' gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = #### gitlab_rails['smtp_port'] = #### gitlab_rails['smtp_user_name'] = #### gitlab_rails['smtp_password'] = #### gitlab_rails['smtp_domain'] = #### gitlab_rails['smtp_authentication'] = "login" gitlab_rails['smtp_enable_starttls_auto'] = true gitlab_rails['smtp_tls'] = false gitlab_rails['smtp_openssl_verify_mode'] = 'peer' gitlab_rails['gitlab_default_projects_features_issues'] = false gitlab_rails['gitlab_default_projects_features_wiki'] = false gitlab_rails['gitlab_default_projects_features_container_registry'] = false