Possible data loss on postgres failover
I noticed that we use the following patroni settings:
bootstrap.dcs.postgresql.use_pg_rewind: true
bootstrap.dcs.synchronous_mode: false # implicit
bootstrap.dcs.maximum_lag_on_failover: 1048576From https://patroni.readthedocs.io/en/latest/replication_modes.html#synchronous-mode:
For use cases where losing committed transactions is not permissible you can turn on Patroni’s synchronous_mode. When synchronous_mode is turned on Patroni will not promote a standby unless it is certain that the standby contains all transactions that may have returned a successful commit status to client
Our settings mean that our cluster is happy to silently lose up to 1MB of writes on failover, using pg_rewind to remove them. Of course this doesn't affect git / registry data, but will cause data loss for all GitLab metadata: users, groups, issues, merge requests, etc. This does not seem like a good state of affairs to me.
Am I missing some historical context on why we choose availability over consistency? If not, then I propose we reverse this situation, and value consistency over availability.
A possible implementation of this could look like:
Configure patroni with:
bootstrap.dcs.postgresql.use_pg_rewind: false
bootstrap.dcs.maximum_lag_on_failover: 0
bootstrap.dcs.synchronous_mode: true
bootstrap.dcs.synchronous_mode_strict: trueAccording to the patroni docs, synchronous_mode_strict ensures that synchronous replication is never disabled, causing writes to be rejected if the primary has not received commit acknowledgement from any standby At the postgres level, we should validate that num_sync in synchronous_standby_names is set to at least 1 (https://www.postgresql.org/docs/9.6/runtime-config-replication.html).