Add HA support for other databases (Praefect, Geo)

Follow up from &7814 (closed)

Support in Omnibus for deploying all databases in HA in Omnibus is not available. In the Reference Architectures the goal would be to support the disparate setups, such as:

• GitLab and Praefect / Geo DB's colocated - Where databases are on the same cluster, connecting through the same PgBouncer array. Note - Doesn't support Geo due to replication requirements. There's also variations of the above, such as Praefect and Geo being colocated but the main GitLab database being separated. There's also the possibility that further databases will be added down the line.

• GitLab and Praefect / Geo DB's separated - Where databases have their own cluster and PgBouncer array. Note - Does support Geo, only the main GitLab DB is replicated.

Today it's not possible in Omnibus to configure this. This is because the watcher and failover scripts in omnibus are only added once, and for a single configurable database.

When colocated, the service, watcher and failover script all effectively are only looking for a single database name and as a result if you attempt to set up another database connection, like praefect_production, it will be ignored:

[databases]

gitlabhq_production = host=gitlab-qa-10k-test-postgres-1.c.gitlab-qa-10k-cd77c7.internal auth_user=pgbouncer dbname=gitlabhq_production

praefect_production =

praefect_production_direct = dbname=praefect_production pool_mode=session

The same also applies when trying to set up a separate database cluster where you effectively need to "hack" the Consul postgresql watcher to be only for the other database by overriding settings such as node['consul']['internal']['postgresql_service_name'] and what looks like node['gitlab']['gitlab-rails']['db_database'] to get it over the line. But the same issue noted above is also at play here as it seems only the database named in node['gitlab']['gitlab-rails']['db_database'] will ever get updated so other connections like the second Praefect cache connection are ignored.