Reconfigure appears to not start Patroni (and Postgres) properly

While reconfiguring a node after dealing with #5746 (closed) I got a timeout start error for Postgres via Patroni:

  * ruby_block[wait for postgresql to start] action run

    ================================================================================
    Error executing action `run` on resource 'ruby_block[wait for postgresql to start]'
    ================================================================================

    Timeout::Error
    --------------
    execution expired

    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:95:in `sleep'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:95:in `block (3 levels) in from_file'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:94:in `block (2 levels) in from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb

     92: ruby_block 'wait for postgresql to start' do
     93:   block do
     94:     Timeout.timeout(30) do
     95:       sleep 2 until pg_helper.ready?
     96:     end
     97:   end
     98: end
     99:

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:92:in `from_file'

    ruby_block("wait for postgresql to start") do
      action [:run]
      default_guard_interpreter :default
      declared_type :ruby_block
      cookbook_name "patroni"
      recipe_name "enable"
      block #<Proc:0x000055e67361d1e0 /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:93>
      block_name "wait for postgresql to start"
    end

    System Info:
    ------------
    chef_version=15.14.0
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:

execution expired

On further investigation it appears that Patroni doesn't seem to be started a part of gitlab-ctl reconfigure (on the latest nightly omnibus package - 13.6.0-pre (400d0718b06)) after it was specifically stopped with gitlab-ctl stop. Starting it again with gitlab-ctl start beforehand and then re-running reconfigure solved the issue.

It wouldn't be too uncommon for users to have patroni stopped beforehand before running reconfigure so this should be addressed.