Skip to content

Timeout error on Geo secondary read-replica near the end of `gitlab-ctl pg-upgrade`

During pg-upgrade command, I got a timeout while Waiting for Database to be running (see #4975 (comment 319485449)):

Running handlers:
Running handlers complete
Chef Client finished, 2/270 resources updated in 04 seconds
Running reconfigure: OK
Waiting for Database to be running.
Traceback (most recent call last):
	14: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
	13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
	12: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
	11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
	10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
	 9: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:187:in `block in load_file'
	 8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:312:in `geo_secondary_upgrade'
	 7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:217:in `common_post_upgrade'
	 6: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:11:in `wait_for_postgresql'
	 5: from /opt/gitlab/embedded/lib/ruby/2.6.0/timeout.rb:108:in `timeout'
	 4: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:12:in `block in wait_for_postgresql'
	 3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:12:in `loop'
	 2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:13:in `block (2 levels) in wait_for_postgresql'
	 1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:18:in `rescue in block (2 levels) in wait_for_postgresql'
/opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:18:in `sleep': execution expired (Timeout::Error)
	9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
	8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
	7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
	6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
	5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
	4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:187:in `block in load_file'
	3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:312:in `geo_secondary_upgrade'
	2: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:217:in `common_post_upgrade'
	1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:9:in `wait_for_postgresql'
/opt/gitlab/embedded/service/omnibus-ctl/lib/postgresql.rb:25:in `rescue in wait_for_postgresql': uninitialized constant Timeout::TimeoutError (NameError)
  • It appears the PG version upgrade was actually successful, and the DB was running. So on it's face it looks like this timeout should be longer.
  • Also, this appears to be an secondary error in the timeout logic uninitialized constant Timeout::TimeoutError (NameError)
Edited by Michael Kozono