Configure pg-upgrade to work with patroni service

What does this MR do?

This MR enables gitlab-ctl pg-upgrade command to work on Patroni cluster nodes. On the leader node it upgrades PostgreSQL and wipes the cluster state from Consul and signal it to reinitialize. On replica nodes it deletes the data directory, upgrades PostgreSQL and joins the cluster. Patroni will use pg_basebackup to resynchronize the replica.

This MR also adds to additional command arguments to pg-upgrade. --leader and --replica will override automatic detection of the node role and change the upgrade procedure accordingly.

This seems to be excessive, esp. because it requires shutting down the cluster during upgrade (at least as long as the leader is upgrading) but it is currently the only way that it is possible, for example see https://github.com/zalando/patroni/issues/541#issuecomment-349229838

NOTE: This MR is blocked by !4548 (merged) and requires its changes to work properly.

NOTE: Documentation is in gitlab!41732 (merged).

Caveats

1. The solution in this MR only works for pg-upgrade. Reverting a successful upgrade with revert-pg-upgrade will be problematic and requires manual reinitialization of the cluster. Given the complexity and risks involved we should not recommend using revert-pg-upgrade and should encourage users to seek help from support.
2. This MR can detect the leader node. However, it can not automatically detect the replica nodes and some of database settings (e.g. locale, collate, etc). Therefore it relies on user parameters that are passed with command arguments (e.g. --replica) or environment variable. Replica detection can be replaced with a heuristic to eliminate the need for parameter passing.

How to test it?

1. Deploy a Patroni cluster (you can use https://gitlab.com/pursultani/gitlab-patroni-test).
2. Follow instructions in gitlab!41732 (merged).
3. PostgreSQL on all cluster nodes must be upgraded and all of them must be healthy.

Related issues

Closes #5423 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

• Merge Request Title, and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
• trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for the GitLab Chart opened

changed milestone to %13.4

added Deliverable devopssystems direction groupdistribution ~28039 priority2 sectioncore platform severity2 typebug labels

mentioned in issue #5423 (closed)

assigned to @pursultani

added 1 commit

• d86c7ea2 - Configure pg-upgrade to work with patroni service

Compare with previous version

	1 Warning
	You’ve made some changes in the cookbooks, but didn’t add any tests. That’s OK as long as you’re refactoring existing code, but please consider adding the ~”feature::maintenance” label in that case.

	1 Message
	Please add the ~”workflow::ready for review” label once you think the MR is ready to for an initial review. Merge requests are handled according to the workflow documented in our handbook and should receive a response within the limit documented in our First-response SLO. If you don’t receive a response, please mention @gitlab-org/distribution, or one of our Project Maintainers

Generated by Danger

added 1 commit

• 9b9d7a0d - Configure pg-upgrade to work with patroni service

Compare with previous version

added 1 commit

• bfdde354 - Configure pg-upgrade to work with patroni service

Compare with previous version

added 1 commit

• 47f848ee - Configure pg-upgrade to work with patroni service

Compare with previous version

added 1 commit

• d8629086 - Adjust pg-upgrade command to work with Patroni cluster

Compare with previous version

mentioned in issue gitlab#239020 (closed)

mentioned in merge request !4548 (merged)

added 1 commit

• 19018ae8 - Adjust pg-upgrade command to work with Patroni cluster

Compare with previous version

added 1 commit

• 4d05e575 - Adjust pg-upgrade command to work with Patroni cluster

Compare with previous version

added 1 commit

• d6bcba17 - Adjust pg-upgrade command to work with Patroni cluster

Compare with previous version

mentioned in merge request gitlab!41732 (merged)

changed the description

added 2 commits

• bcc82e40 - Add changelog entry for this MR
• 911c0e84 - Refer to the PostgreSQL upgrade in EE documentation

Compare with previous version

unmarked as a Work In Progress

added workflowready for review label