### Summary Omnibus GitLab uses Repmgr 3 to handle failover for PostgreSQL in self-managed instances that have HA configured. For reasons described below, we need to either update the version of Repmgr or switch to a different solution. After evaluating various options, we have decided to move to Patroni. ### Problems to solve * We have a plan to move to PostgreSQL 11 and 12 during 2020 so that the GitLab community can benefit from performance improvements released in those PostgreSQL versions. Repmgr 3 is no longer well supported. Documentation is no longer available and it is not included in the support matrix for PostgreSQL 11. Although our testing has proven that Repmgr 3 does work with PostgreSQL 11, we prefer to move to a failover solution that is well supported with the latest versions of PostgreSQL. * Repmgr 4 is basically an entirely new product that would require a lot of effort to support. Most of the commands that GitLab users are accustomed to using would change or need to be translated to support an upgrade from Repmgr 3 to Repmgr 4. Even if we put in the time and effort to translate these commands, an upgrade from Repmgr 3 to Repmgr 4 would be a manual upgrade for our users because nodes need to be remapped. * There were [some significant changes to the configuration of replication in PostgreSQL 12](https://www.2ndquadrant.com/en/blog/replication-configuration-changes-in-postgresql-12/). Neither Repmgr 3 or Repmgr 4 work with PostgreSQL 12. We either need to upgrade to Repmgr 5 or switch to Patroni before we can add support for PostgreSQL 12. Patroni 1.6.0 added support for PostgreSQL 12. * Relying on repmgr in Omnibus GitLab while using Patroni in gitlab.com is [a dogfooding anti-pattern that we have committed to avoiding](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/5058). * There have been two reports of brain split issues with repmgr (https://gitlab.com/gitlab-com/migration/issues/770) and we don't think these are resolved by upgrading to Repmgr 4. * We have had a request from a Geo customer to support cascading replication in the Geo secondary cluster. Repmgr 3.x does not support cascading replication. Will switching to Patroni unblock the Geo team? (@fzimmer found [this Patroni documentation](https://patroni.readthedocs.io/en/latest/replica_bootstrap.html#standby-cluster). It looks like what we need.) ### Proposal * In GitLab 13.x, add support for Patroni while continuing to support Repmgr 3.x. At the time of writing, Patroni 1.6.3 is the latest version. Patroni 1.6.0 added support for PostgreSQL 12. Given [our current plans for updating PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2184) and [our plans for releasing major versions of GitLab](https://docs.gitlab.com/ee/policy/maintenance.html), bundling Patroni 1.6.x would put us in a good position until at least May 2021. * Remove Repmgr in GitLab 14.0 and require Patroni. This gives us 12 months to confirm that Patroni is performing as expected before making it a requirement. * Create clear user documentation that guides users through a manual switch from Repmgr to Patroni while minimizing the impact on availability. ### Acceptance Criteria * [ ] Replication and failover with Patroni has been tested at different scales and performance is the same or better compared to Repmgr * [ ] Downtime when switching from Repmgr to Patroni is minimized as much as possible * [ ] Migration guidelines exist that help users minimize downtime even further * [ ] Reuse what we can from setting up Patroni for gitlab.com * [x] Patroni works correctly with PostgreSQL 11 and 12 * [x] Patroni works correctly with Geo * [ ] The issue https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7790 has been reviewed and is not considered to be a concern for Omnibus (or it guides us on the default configurations that we should ship with) * [ ] Users do not lose any HA functionality by switching from Repmgr to Patroni * [ ] Documentation includes guidelines on tuning failure detection so that a user can balance false positives with detection time ### User value * Users can use PostgreSQL 12 with a GitLab-provided HA solution, benefitting from the performance improvements that PostgreSQL offers * The Geo team can support cascading replication * The replication and failover solution used for newer versions of PostgreSQL is fully supported ### Steps * [ ] An MR to add Patroni to Omnibus was started here https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3196. Start by reviewing this MR, determining if it still makes sense to continue with it, and bring it up to date given the time that has lapsed since it was last worked on and unresolved comments * [ ] Add appropriate commands to implement required operational steps (like the gitlab-ctl repmgr commands) * [ ] Determine whether a load balancer is required for Patroni. If yes, create the necessary issues. ([This page](https://about.gitlab.com/handbook/engineering/infrastructure/library/postgres-ha/#gitlabcom-and-self-managed) may be a good starting point) * [ ] Create a test plan * [ ] Test * [ ] Create user documentation on how to switch from repmgr to Patroni * [ ] Create user documentation on configuring and using Patroni * [ ] Implement guardrails to prevent both repmgr and Patroni from being enabled at the same time (?) * [ ] Add support for Patroni to the Provisioner tool for automating HA/Geo deployments ### Notes * A lengthy discussion about switching to Patroni is in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3752 * Geo epic for PostgreSQL HA on the secondary cluster: https://gitlab.com/groups/gitlab-org/-/epics/2536