Treat Patroni streaming state as a running node

What does this MR do?

Patroni 3.0.4+ reports the state of a healthy standby as streaming (when it has a live replication connection) or in archive recovery (when it is replaying from the WAL archive) instead of the generic running that the leader reports. See the Patroni 3.0.4 release notes. This behavior was introduced by patroni#2733 (d46ca88e), first shipped in Patroni 3.0.4.

The patroni cookbook and gitlab-ctl pg-upgrade only recognized running, so on any cluster running Patroni 3.0.4 or newer every healthy replica looked "not running". This shipped in GitLab 18.9.0, which bumped Patroni from 3.0.1 to 3.3.1. On a replica:

  • the wait for node bootstrap to complete loop never saw running and burned its full 30s timeout on every gitlab-ctl reconfigure;
  • the edit-config, reload, and restart guards in patroni::enable silently skipped;
  • pg-upgrade stopped warning that replicas were still running before a major upgrade.

This adds PatroniHelper#node_running?, which treats running, streaming, and in archive recovery as healthy, and uses it for the recipe guards and the wait loop. The same set of states is applied to the replica count in gitlab-ctl pg-upgrade.

Investigation for #9454

How this was tested

Automated

  • New PatroniHelper#node_running? specs cover running, streaming, and in archive recovery (healthy) plus stopped (not healthy).
  • A patroni::enable recipe spec asserts the restart still fires when a node reports streaming.
  • bundle exec rspec (patroni cookbook + gitlab_ctl/patroni) and bundle exec rubocop pass locally.

End-to-end on a real cluster

Stood up a 3-node Patroni cluster on GCP: 3 Ubuntu 22.04 VMs, each running Consul (server, quorum of 3) + Patroni + PostgreSQL, roles(['patroni_role']). Installed GitLab EE 19.3.2 (patronictl version reports 3.3.1, which has the streaming behavior and the hardcoded node_status == 'running' guards).

Cluster state — healthy replicas report streaming, the leader reports running:

| Member       | Role    | State     |
| sh-patroni-1 | Replica | streaming |
| sh-patroni-2 | Leader  | running   |
| sh-patroni-3 | Replica | streaming |

Reproduced the bug (unpatched code). On replica sh-patroni-1, changed a DCS setting (patroni['loop_wait'] 10 → 20) and ran gitlab-ctl reconfigure. node_status returns streaming, so the only_if guard fails:

* execute[update dynamic configuration settings] action run (skipped due to only_if)

The local dcs.yaml regenerated to loop_wait: 20, but the cluster DCS stayed at loop_wait: 10 (patronictl show-config). The replica's config change silently never reached the cluster.

Verified the fix. Applied this MR's changes to the installed patroni_helper.rb and patroni/recipes/enable.rb on the same node (in place, to avoid a full package rebuild; identical Ruby to the diff here), then reconfigured with patroni['loop_wait'] → 30:

* execute[update dynamic configuration settings] action run          # no longer skipped
* ruby_block[wait for node bootstrap to complete] action run         # returns immediately, no 30s timeout
loop_wait: 30                                                         # cluster DCS updated

gitlab-ctl pg-upgrade replica count. Ran the counting logic against the live /cluster REST data:

running_replica_count OLD (member state == 'running'): 0   # misses both streaming replicas
running_replica_count NEW (RUNNING_STATES.include?):   2   # correct

Without the fix, the "shut down all replicas before upgrading" safety warning never fires. The VMs were deleted after testing.

Checklist

  • Tests added/updated (patroni_helper_spec.rb, patroni_spec.rb)
  • bundle exec rspec (patroni specs) and bundle exec rubocop pass locally
  • Validated end-to-end on a real 3-node Patroni cluster (GitLab EE 19.3.2, Patroni 3.3.1); see the validation results in the discussion below
  • Apply the workflowready for review label
Edited by Stan Hu

Merge request reports

Loading
Loading