Skip to content

gitlab-ctl reconfigure fails due to slow postgres start up

First I aplogise if this is a duplicate I tried searching but keep getting 500 errors.

Summary

gitlab-ctl reconfigure fails when postgres is still starting up with the error:

There was an error running gitlab-ctl reconfigure:                                                  
                                                                                                    
postgresql_user[gitlab] (postgresql::enable line 196) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[create gitlab postgresql user] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb line 11) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
---- Begin output of /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\"" ----      
STDOUT:                                                                                             
STDERR: psql: FATAL:  the database system is starting up                                            
---- End output of /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\"" ----        
Ran /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\"" returned 2

Steps to reproduce

I encountered this on a server running Ubuntu 18.04 with a software raid 5 array with spinning disk. The checkarray script was running which checks the array, with it being raid 5 this makes the disk responsiveness awful. During the check gitlab which runs via the gitlab/gitlab-ce image was being restarted and then generated the error.

Obviously this isn't the easiest to replicate, I'm happy to re-trigger checkarray to test any changes.

What is the current bug behavior?

During startup gitlab-ctl reconfigure fails

What is the expected correct behavior?

gitlab-ctl should wait for postgres to be working before executing migrations

Relevant logs

See full log file

gitlab.log

Details of package version

11.8.0-ce.0 via docker

Environment details

  • Operating System: Ubuntu 18.04
  • Installation Target, remove incorrect values:
    • Bare Metal Machine
  • Installation Type, remove incorrect values:
    • New Installation
  • Is there any other software running on the machine: N/A
  • Is this a single or multiple node installation? Single
  • Resources
    • CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
    • Memory total: 32GB