Skip to content

Configure pg-upgrade to work with patroni service

Hossein Pursultani requested to merge 5423-patroni-pg-upgrade-fix into master

What does this MR do?

This MR enables gitlab-ctl pg-upgrade command to work on Patroni cluster nodes. On the leader node it upgrades PostgreSQL and wipes the cluster state from Consul and signal it to reinitialize. On replica nodes it deletes the data directory, upgrades PostgreSQL and joins the cluster. Patroni will use pg_basebackup to resynchronize the replica.

This MR also adds to additional command arguments to pg-upgrade. --leader and --replica will override automatic detection of the node role and change the upgrade procedure accordingly.

This seems to be excessive, esp. because it requires shutting down the cluster during upgrade (at least as long as the leader is upgrading) but it is currently the only way that it is possible, for example see https://github.com/zalando/patroni/issues/541#issuecomment-349229838

NOTE: This MR is blocked by !4548 (merged) and requires its changes to work properly.

NOTE: Documentation is in gitlab!41732 (merged).

Caveats

  1. The solution in this MR only works for pg-upgrade. Reverting a successful upgrade with revert-pg-upgrade will be problematic and requires manual reinitialization of the cluster. Given the complexity and risks involved we should not recommend using revert-pg-upgrade and should encourage users to seek help from support.
  2. This MR can detect the leader node. However, it can not automatically detect the replica nodes and some of database settings (e.g. locale, collate, etc). Therefore it relies on user parameters that are passed with command arguments (e.g. --replica) or environment variable. Replica detection can be replaced with a heuristic to eliminate the need for parameter passing.

How to test it?

  1. Deploy a Patroni cluster (you can use https://gitlab.com/pursultani/gitlab-patroni-test).
  2. Follow instructions in gitlab!41732 (merged).
  3. PostgreSQL on all cluster nodes must be upgraded and all of them must be healthy.

Related issues

Closes #5423 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
  • trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Tests added
  • Integration tests added to GitLab QA
  • Equivalent MR/issue for the GitLab Chart opened
Edited by Hossein Pursultani

Merge request reports