<!--- Please read this! Before you create a new issue, please make sure that you already ran `sudo gitlab-ctl reconfigure` Also make sure to search in https://gitlab.com/gitlab-org/omnibus-gitlab/issues, to verify the issue you're about to submit isn't a duplicate. ---> ### Summary I'm checking a customer's procedure for setting up a Patroni cluster with the `gitlab_replicator` password in place, and switching to `md5_auth_cidr_addresses` GitLab team members can read more [in the ticket](https://gitlab.zendesk.com/agent/tickets/243196). Customer has specified `patroni['replication_password']` in addition to `postgresql['sql_replication_password']` Having followed the procedure and switched to `md5_auth_cidr_addresses` it seems that `patroni['replication_password']` isn't optional - without this, the replicas fails to connect to the leader - specifically patroni (it's a python stack trace .. [see below](relevant-logs) ``` FATAL: could not connect to the primary server: fe_sendauth: no password supplied ERROR: Can not fetch local timeline and lsn from replication connection [..] psycopg2.OperationalError: fe_sendauth: no password supplied ``` [Setting the Patroni password is not in the reference architecture docs](https://docs.gitlab.com/ee/administration/reference_architectures/10k_users.html#configure-postgresql). My working assumption is that `patroni['replication_password']` is required, at least when using `md5_auth_cidr_addresses` in which case it's a docs. #### bootstrapping issue In issue [Possible infinite loop with Patroni bootstrapping a follower ...](https://gitlab.com/gitlab-org/gitlab/-/issues/246545#note_445078301): > 1. A patroni cluster (3 nodes) > 1. Each patroni node should not have "trust" authentication (it should use MD5 in order to ask for passwords) > 1. You shouldn't configure `patroni['replication_password']` See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6590#note_783756802 for more. ### Steps to reproduce Set up a patroni cluster with a `gitlab_replicator` password and switch from `trust_auth_cidr_addresses` to `md5_auth_cidr_addresses` ### What is the current *bug* behavior? replias cannot connect to leader ### What is the expected *correct* behavior? replicas connect to leader ### Relevant logs <details> <summary> Relevant logs </summary> <pre> 2021-12-17_12:40:43.01716 FATAL: could not connect to the primary server: fe_sendauth: no password supplied 2021-12-17_12:40:43.99269 2021-12-17 12:40:43,984 ERROR: Can not fetch local timeline and lsn from replication connection 2021-12-17_12:40:43.99272 Traceback (most recent call last): 2021-12-17_12:40:43.99273 File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/postgresql/__init__.py", line 737, in get_replica_timeline 2021-12-17_12:40:43.99273 with self.get_replication_connection_cursor(**self.config.local_replication_address) as cur: 2021-12-17_12:40:43.99274 File "/opt/gitlab/embedded/lib/python3.7/contextlib.py", line 112, in __enter__ 2021-12-17_12:40:43.99274 return next(self.gen) 2021-12-17_12:40:43.99275 File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/postgresql/__init__.py", line 732, in get_replication_connection_cursor 2021-12-17_12:40:43.99275 with get_connection_cursor(**conn_kwargs) as cur: 2021-12-17_12:40:43.99276 File "/opt/gitlab/embedded/lib/python3.7/contextlib.py", line 112, in __enter__ 2021-12-17_12:40:43.99276 return next(self.gen) 2021-12-17_12:40:43.99277 File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/postgresql/connection.py", line 43, in get_connection_cursor 2021-12-17_12:40:43.99277 with psycopg2.connect(**kwargs) as conn: 2021-12-17_12:40:43.99278 File "/opt/gitlab/embedded/lib/python3.7/site-packages/psycopg2-2.8.6-py3.7-linux-x86_64.egg/psycopg2/__init__.py", line 127, in connect 2021-12-17_12:40:43.99281 conn = _connect(dsn, connection_factory=connection_factory, **kwasync) 2021-12-17_12:40:43.99281 psycopg2.OperationalError: fe_sendauth: no password supplied </pre> </details> ### Details of package version <details> <summary>Provide the package version installation details</summary> <pre> <!-- For Debian, Ubuntu based systems, paste the output of: `dpkg-query -l "gitlab-*"` For RHEL based systems (CentOS, RHEL, OL, Scientific, OpenSUSE, SLES), paste the output of: `rpm -qa | grep 'gitlab'` If you can't provide these details, see https://about.gitlab.com/getting-help/ page on where to ask your question. --> </pre> </details> ### Environment details Patroni cluster built using 13.12.15 in docker ### Configuration details Part way through a variety of tests, so at this time the cluster's running PostgreSQL 11. But I don't think is relevant. <details> <summary> Provide the relevant sections of `/etc/gitlab/gitlab.rb` </summary> <pre> roles ['postgres_role'] consul['configuration'] = { retry_join: ['172.18.10.101:8301','172.18.10.102:8301','172.18.10.103:8301'], server: false, } # Enable Patroni (which automatically disables Repmgr). patroni['enable'] = true # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' # added loopback (not in our docs for Patroni) postgresql['md5_auth_cidr_addresses'] = ['172.18.10.0/24','127.0.0.1/32'] # Configure the Consul agent consul['services'] = ['postgresql'] postgresql['pgbouncer_user_password'] = File.open('/etc/gitlab/secrets/pg_pgbouncer_user_password').first.chomp postgresql['sql_user_password'] = File.open('/etc/gitlab/secrets/pg_sql_user_password').first.chomp # to prevent replication from using up all of the available database connections # set to one more than the number of PG servers patroni['postgresql']['max_wal_senders'] = 4 patroni['postgresql']['max_replication_slots'] = 4 # disable prometheus prometheus_monitoring['enable'] = false # logrotate not needed, sv handles consul logrotate['enable'] = false # Disable auto migrations gitlab_rails['auto_migrate'] = false postgresql['version'] = 11 postgresql['sql_replication_password'] = File.open('/etc/gitlab/secrets/pg_sql_repl_password').first.chomp patroni['replication_password'] = File.open('/etc/gitlab/secrets/patroni_repl_password').first.chomp postgresql['sql_replication_user'] = 'gitlab_replicator' consul['logging_filters'] = {} </pre> </details>