Refresh Load Balancer (Amphora) configuration when Octavia Health Manager IP addresses change

Summary

When a Load Balancer VM ("Amphora") is created by Octavia, a static configuration is passed to it. This configuration contains a list of IP addresses of the Octavia Health Managers. Amphorae will send out periodic heartbeats to all of these IP addresses.

Due to the rework in !2926 (merged) the scaling and distribution of the Health Manager Pods may change at any point, making the list of addresses not static anymore. As as a result, old Health Manager IP addresses might become invalid/unreachable and new ones might be added.

The Amphorae's configuration is not updated at runtime and once no functional Health Manager address is left in their configuration, their heartbeats will stop being received, they will be declared as offline by the Health Managers and failovered automatically.

Use cases

The set of Health Manager IP addresses may change at any time:

  • the Octavia control plane label is removed from a Kubernetes node for downscaling or maintenance purposes
  • the Octavia control plane label is added to a Kubernetes node for the purpose of scaling up the Octavia control plane replicas
  • due to a disruption or node failure, Octavia Pods are reinitialized and get new Health Manager addresses assigned

Proposal

The octavia-operator should implement mechanisms for updating the configuration of existing Amphorae once the set of Health Manager IPs change.

The Octavia LBaaS API offers an Amphora update call that might help:

Update the amphora agent configuration. This will push the new configuration to the amphora agent and will update the configuration options that are mutatable.

See also python-octaviaclient/octaviaclient/api/v2/octavia.py:amphora_configure().

Specification

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this issue are to be interpreted in the spirit of RFC 2119, even though we're not technically doing protocol design.