Enabling Geo secondary starts the geo-postgresql database on the wrong port

Steps to reproduce:

  1. Download @brodock's latest image.
  2. Install it.
  3. Add geo_secondary_role['enable'] = true to /etc/gitlab/gitlab.rb
  4. Run gitlab-ctl reconfigure

It appears that there is a race condition occurring here. The runit service definition creates a new service, which is automatically started. That service spins until a postgresql.conf file is available, which happens with the initdb call.

Attached are the logs:

reconfigure_fail.txt

stanhu@geo-secondary-test:/tmp$ sudo tail /var/log/gitlab/geo-postgresql/current
2017-04-08_14:05:38.29090 postgres: could not access the server configuration file "/var/opt/gitlab/geo-postgresql/data/postgresql.conf": No such file or directory
2017-04-08_14:05:39.30951 FATAL:  lock file "postmaster.pid" already exists
2017-04-08_14:05:39.30952 HINT:  Is another postgres (PID 10035) running in data directory "/var/opt/gitlab/geo-postgresql/data"?
2017-04-08_14:05:40.33141 FATAL:  lock file "postmaster.pid" already exists
2017-04-08_14:05:40.33143 HINT:  Is another postgres (PID 10040) running in data directory "/var/opt/gitlab/geo-postgresql/data"?
2017-04-08_14:05:41.92292 LOG:  database system was shut down at 2017-04-08 14:05:41 UTC
2017-04-08_14:05:41.92999 LOG:  MultiXact member wraparound protections are now enabled
2017-04-08_14:05:41.93070 LOG:  database system is ready to accept connections
2017-04-08_14:05:41.93328 LOG:  autovacuum launcher started
stanhu@geo-secondary-test:/tmp$ sudo ls -al /var/opt/gitlab/geo-postgresql/data/postgresql.conf  
-rw-r--r-- 1 gitlab-psql root 20142 Apr  8 14:05 /var/opt/gitlab/geo-postgresql/data/postgresql.conf
stanhu@geo-secondary-test:/tmp$ sudo gitlab-ctl status
run: geo-postgresql: (pid 10044) 390s; run: log: (pid 10021) 393s
run: gitaly: (pid 10600) 310s; run: log: (pid 9860) 436s
run: gitlab-monitor: (pid 10645) 308s; run: log: (pid 9996) 399s
run: gitlab-workhorse: (pid 10611) 309s; run: log: (pid 9871) 430s
run: logrotate: (pid 9896) 421s; run: log: (pid 9895) 421s
run: nginx: (pid 9882) 427s; run: log: (pid 9881) 427s
run: node-exporter: (pid 9945) 413s; run: log: (pid 9944) 413s
run: postgres-exporter: (pid 10634) 308s; run: log: (pid 9969) 405s
run: postgresql: (pid 9740) 450s; run: log: (pid 9739) 450s
run: prometheus: (pid 10622) 309s; run: log: (pid 9928) 419s
run: redis: (pid 9672) 456s; run: log: (pid 9671) 456s
run: redis-exporter: (pid 9957) 411s; run: log: (pid 9956) 411s
run: sidekiq: (pid 11345) 12s; run: log: (pid 9850) 438s
run: unicorn: (pid 9820) 444s; run: log: (pid 9819) 444s
stanhu@geo-secondary-test:/tmp$ ps -ef | grep geo
root     10020  9639  0 14:05 ?        00:00:00 runsv geo-postgresql
root     10021 10020  0 14:05 ?        00:00:00 svlogd -tt /var/log/gitlab/geo-postgresql
gitlab-+ 10044 10020  0 14:05 ?        00:00:00 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/geo-postgresql/data
stanhu   11493  1904  0 14:12 pts/0    00:00:00 grep --color=auto geo