Mattermost-only server manual gitlab-ctl pg-upgrade run failure: Waiting for Database to be running.
Got this error on server running only Mattermost:
...
Running handlers:
Running handlers complete
Chef Client finished, 4/411 resources updated in 22 seconds
Running reconfigure: OK
Waiting for Database to be running.
Traceback (most recent call last):
14: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
12: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
9: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:166:in `block in load_file'
8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:221:in `general_upgrade'
7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:195:in `common_post_upgrade'
6: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:11:in `wait_for_postgresql'
5: from /opt/gitlab/embedded/lib/ruby/2.6.0/timeout.rb:108:in `timeout'
4: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:12:in `block in wait_for_postgresql'
3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:12:in `loop'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:13:in `block (2 levels) in wait_for_postgresql'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:18:in `rescue in block (2 levels) in wait_for_postgresql'
/opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:18:in `sleep': execution expired (Timeout::Error)
9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:166:in `block in load_file'
3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:221:in `general_upgrade'
2: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:195:in `common_post_upgrade'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:9:in `wait_for_postgresql'
/opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:25:in `rescue in wait_for_postgresql': uninitialized constant Timeout::TimeoutError (NameError)
Fixed with:
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
sudo gitlab-ctl deploy-page down
sudo gitlab-ctl reconfigure
13.11.3 fixed with:
- Need to run
ANALYZE
manually?sudo gitlab-psql -d mattermost_production -c "SELECT relname, last_analyze, last_autoanalyze FROM pg_stat_user_tables WHERE last_analyze IS NULL AND last_autoanalyze IS NULL;"
- If anything returned run:
sudo gitlab-psql -d mattermost_production -c 'SET statement_timeout = 0; ANALYZE VERBOSE;'
- If anything returned run:
sudo gitlab-ctl deploy-page down
sudo gitlab-ctl start
- working at this point but ran below anyways
-
sudo gitlab-ctl reconfigure
(looks like this didn't do anything meaningful)
Edited by Ben Bodenmiller