Treat Patroni streaming state as a running node
What does this MR do?
Patroni 3.0.4+ reports the state of a healthy standby as streaming
(when it has a live replication connection) or in archive recovery
(when it is replaying from the WAL archive) instead of the generic
running that the leader reports. See the
Patroni 3.0.4 release notes. This behavior was introduced by patroni#2733 (d46ca88e), first shipped in Patroni 3.0.4.
The patroni cookbook and gitlab-ctl pg-upgrade only recognized
running, so on any cluster running Patroni 3.0.4 or newer every healthy
replica looked "not running". This shipped in GitLab 18.9.0, which
bumped Patroni from 3.0.1 to 3.3.1. On a replica:
- the
wait for node bootstrap to completeloop never sawrunningand burned its full 30s timeout on everygitlab-ctl reconfigure; - the
edit-config,reload, andrestartguards inpatroni::enablesilently skipped; pg-upgradestopped warning that replicas were still running before a major upgrade.
This adds PatroniHelper#node_running?, which treats running,
streaming, and in archive recovery as healthy, and uses it for the
recipe guards and the wait loop. The same set of states is applied to the
replica count in gitlab-ctl pg-upgrade.
Related issues
Investigation for #9454
How this was tested
Automated
- New
PatroniHelper#node_running?specs coverrunning,streaming, andin archive recovery(healthy) plusstopped(not healthy). - A
patroni::enablerecipe spec asserts the restart still fires when a node reportsstreaming. bundle exec rspec(patroni cookbook +gitlab_ctl/patroni) andbundle exec rubocoppass locally.
End-to-end on a real cluster
Stood up a 3-node Patroni cluster on GCP: 3 Ubuntu 22.04 VMs, each running
Consul (server, quorum of 3) + Patroni + PostgreSQL, roles(['patroni_role']).
Installed GitLab EE 19.3.2 (patronictl version reports 3.3.1, which
has the streaming behavior and the hardcoded node_status == 'running'
guards).
Cluster state — healthy replicas report streaming, the leader reports
running:
| Member | Role | State |
| sh-patroni-1 | Replica | streaming |
| sh-patroni-2 | Leader | running |
| sh-patroni-3 | Replica | streaming |Reproduced the bug (unpatched code). On replica sh-patroni-1, changed a
DCS setting (patroni['loop_wait'] 10 → 20) and ran gitlab-ctl reconfigure.
node_status returns streaming, so the only_if guard fails:
* execute[update dynamic configuration settings] action run (skipped due to only_if)The local dcs.yaml regenerated to loop_wait: 20, but the cluster DCS stayed
at loop_wait: 10 (patronictl show-config). The replica's config change
silently never reached the cluster.
Verified the fix. Applied this MR's changes to the installed
patroni_helper.rb and patroni/recipes/enable.rb on the same node (in place,
to avoid a full package rebuild; identical Ruby to the diff here), then
reconfigured with patroni['loop_wait'] → 30:
* execute[update dynamic configuration settings] action run # no longer skipped
* ruby_block[wait for node bootstrap to complete] action run # returns immediately, no 30s timeout
loop_wait: 30 # cluster DCS updatedgitlab-ctl pg-upgrade replica count. Ran the counting logic against the
live /cluster REST data:
running_replica_count OLD (member state == 'running'): 0 # misses both streaming replicas
running_replica_count NEW (RUNNING_STATES.include?): 2 # correctWithout the fix, the "shut down all replicas before upgrading" safety warning never fires. The VMs were deleted after testing.
Checklist
- Tests added/updated (
patroni_helper_spec.rb,patroni_spec.rb) -
bundle exec rspec(patroni specs) andbundle exec rubocoppass locally - Validated end-to-end on a real 3-node Patroni cluster (GitLab EE 19.3.2, Patroni 3.3.1); see the validation results in the discussion below
- Apply the workflowready for review label