Engineering Owner: @brodock ### Problem to solve <!-- What problem do we solve? --> The Geo Primary currently supports a high-availability configuration of PostgreSQL [using the PostgreSQL version shipped with Omnibus](https://docs.gitlab.com/ee/administration/high_availability/database.html) or using a different PostgreSQL deployment. Geo Secondaries, however, can currently not be configured in a high-availability configuration. This is problematic if Geo is used as part of a disaster recovery strategy because it means that a failover to a designated secondary can't utilise a high-availability configuration immediately. Exactly mirroring a Primary and a Secondary in deployment is not possible. ### Intended users * [Sidney (Systems Administrator)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator) ### Further details <!-- Include use cases, benefits, and/or goals (contributes to our vision?) --> The reason for the absence of this configuration is that currently `repmgr` in the version that it is shipped with Omnibus [does not work in a read-only node](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3361). In order to support high-availability configurations, enabling [some form of cascading replication](https://gitlab.com/gitlab-org/gitlab/issues/9723), or [supporting several databases](https://gitlab.com/gitlab-org/gitlab/issues/5494) may be desirable. PostgreSQL HA is also using different tools on Gitlab.com and in self-managed instances. GitLab.com uses `Patroni`, whereas omnibus packages only ships with `repmgr`. There is [a discussion to replace `repmgr` with `patroni`](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3752). ### Proposal <!-- How are we going to solve the problem? Try to include the user journey! https://about.gitlab.com/handbook/journeys/#user-journey --> I believe in order to tackle the use case of having an HA PostgreSQL setup available in the Primary and a Secondary Geo node we should proceed as follows from a Geo perspective: 1. Setup an HA Geo cluster following a reference architecture for Xk users 1. Configure the cluster to use an *external* PostgreSQL DB in the following scenario 1. ~~HA PostgreSQL on the Primary using `repmgr` and on the secondary using `repmgr`. This requires using the latest version of `repmgr`. See https://gitlab.com/gitlab-org/gitlab/issues/9723#note_277053426~~ **UPDATE**: Patroni will be added to Omnibus https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5112 and we should not go forward with `repmgr` 1. HA PostgrSQL on the Primary using `patroni` and on the secondary using `patroni`. 1. Evaluate how HA Postgres would work with `patroni` We should give users also the option to include PgBouncer in their setup. See https://docs.gitlab.com/ee/administration/high_availability/database.html#architecture ### Documentation <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html Add all known Documentation Requirements here, per https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements If this feature requires changing permissions, this document https://docs.gitlab.com/ee/user/permissions.html must be updated accordingly. --> This will require significant amounts of documentation ### Availability & Testing Significant amounts of testing are needed, including migrations ### What does success look like, and how can we measure that? <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. --> 1. We can configure a Geo secondary to mirror the primary DB setup with Patroni in a reference architecture ### What is the type of buyer? <!-- Which leads to: in which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers --> * Ultimate * Premium ### Customer interest * https://gitlab.zendesk.com/agent/tickets/113919 * https://gitlab.zendesk.com/agent/tickets/93458 * https://gitlab.zendesk.com/agent/tickets/95357 * https://gitlab.zendesk.com/agent/tickets/131085 * https://gitlab.zendesk.com/agent/tickets/132692 * https://gitlab.zendesk.com/agent/tickets/121892 ### Links / references https://gitlab.com/groups/gitlab-org/-/epics/2588