Support RabbitMQ upgrades

Proposal

1. Ensure quorum is still met while pod restarts and there are no ongoing sync operations

Run

rabbitmq-upgrade await_online_quorum_plus_one

inside preStop. This does not only affect upgrades but restarts in general.

To check sync operations, we can use

rabbitmq-diagnostics check_if_new_quorum_queue_replicas_have_finished_initial_sync

for quorum queues. https://www.rabbitmq.com/docs/upgrade#rolling-upgrade

2. Add a RabbitMQ pre upgrade job

The AMQP policies must be updated before the job is created to ensure that deprecated features like the "classic_queue_mirroring" feature are disabled.

The job needs to run the following and must be set to status "Error" if the pod still uses deprecated features.

rabbitmqctl enable_feature_flag all
rabbitmq-diagnostics check_if_any_deprecated_features_are_used

https://www.rabbitmq.com/docs/rolling-upgrade#verify-all-stable-feature-flags-are-enabled

If possible use a reusable job template for AMQP CLI command that supports both commands that use inter node communication (e.g rabbitmqctl) and rabbitmqadmin, which uses HTTPS and OAuth and pass the commands as parameter.

3. Add a RabbitMQ post upgrade job

Must run

rabbitmqctl enable_feature_flag all
rabbitmq-queues rebalance all

https://www.rabbitmq.com/docs/rolling-upgrade#enable-ff-after-upgrade https://www.rabbitmq.com/docs/upgrade#rebalance

4. Support subsequent upgrades for minor versions

note that only upgrading from one minor to the next minor or major is supported https://www.rabbitmq.com/docs/feature-flags#version-compatibility

The implementation will probably be similiar to the one of the MySQLService upgrade.

Specification

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this issue are to be interpreted in the spirit of RFC 2119, even though we're not technically doing protocol design.