determine requirement for patroni['replication_password']
Summary
I'm checking a customer's procedure for setting up a Patroni cluster with the gitlab_replicator
password in place, and switching to md5_auth_cidr_addresses
GitLab team members can read more in the ticket.
Customer has specified patroni['replication_password']
in addition to postgresql['sql_replication_password']
Having followed the procedure and switched to md5_auth_cidr_addresses
it seems that patroni['replication_password']
isn't optional - without this, the replicas fails to connect to the leader - specifically patroni (it's a python stack trace .. see below
FATAL: could not connect to the primary server: fe_sendauth: no password supplied
ERROR: Can not fetch local timeline and lsn from replication connection
[..]
psycopg2.OperationalError: fe_sendauth: no password supplied
Setting the Patroni password is not in the reference architecture docs.
My working assumption is that patroni['replication_password']
is required, at least when using md5_auth_cidr_addresses
in which case it's a docs.
bootstrapping issue
In issue Possible infinite loop with Patroni bootstrapping a follower ...:
- A patroni cluster (3 nodes)
- Each patroni node should not have "trust" authentication (it should use MD5 in order to ask for passwords)
- You shouldn't configure
patroni['replication_password']
See #6590 (comment 783756802) for more.
Steps to reproduce
Set up a patroni cluster with a gitlab_replicator
password and switch from trust_auth_cidr_addresses
to md5_auth_cidr_addresses
What is the current bug behavior?
replias cannot connect to leader
What is the expected correct behavior?
replicas connect to leader
Relevant logs
Relevant logs
2021-12-17_12:40:43.01716 FATAL: could not connect to the primary server: fe_sendauth: no password supplied 2021-12-17_12:40:43.99269 2021-12-17 12:40:43,984 ERROR: Can not fetch local timeline and lsn from replication connection 2021-12-17_12:40:43.99272 Traceback (most recent call last): 2021-12-17_12:40:43.99273 File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/postgresql/__init__.py", line 737, in get_replica_timeline 2021-12-17_12:40:43.99273 with self.get_replication_connection_cursor(**self.config.local_replication_address) as cur: 2021-12-17_12:40:43.99274 File "/opt/gitlab/embedded/lib/python3.7/contextlib.py", line 112, in __enter__ 2021-12-17_12:40:43.99274 return next(self.gen) 2021-12-17_12:40:43.99275 File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/postgresql/__init__.py", line 732, in get_replication_connection_cursor 2021-12-17_12:40:43.99275 with get_connection_cursor(**conn_kwargs) as cur: 2021-12-17_12:40:43.99276 File "/opt/gitlab/embedded/lib/python3.7/contextlib.py", line 112, in __enter__ 2021-12-17_12:40:43.99276 return next(self.gen) 2021-12-17_12:40:43.99277 File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/postgresql/connection.py", line 43, in get_connection_cursor 2021-12-17_12:40:43.99277 with psycopg2.connect(**kwargs) as conn: 2021-12-17_12:40:43.99278 File "/opt/gitlab/embedded/lib/python3.7/site-packages/psycopg2-2.8.6-py3.7-linux-x86_64.egg/psycopg2/__init__.py", line 127, in connect 2021-12-17_12:40:43.99281 conn = _connect(dsn, connection_factory=connection_factory, **kwasync) 2021-12-17_12:40:43.99281 psycopg2.OperationalError: fe_sendauth: no password supplied