Octavia: MTU-related communication problem between amphora and health-manager.

Environmental conditions

In OpenStack Octavia, the "Health Manager" component monitors the correct functioning of active Layer 7 load balancers. Such a load balancer is implemented, among other things, by one or two Amphora Virtual Machines, which contain the load balancer itself (Haproxy) and an Amphora-Agent.

This agent regularly informs the Health Manager about the status of the active load balancer via UDP communication.

The Amphora Management Network, which is created by the Octavia operator, is used for this purpose. The MTU of this network is currently not set actively, but is automatically configured depending on the MTU of the underlay network (minus Geneve overhead).

On the node where the Health Manager is to be active, an OpenStack Neutron port is created on the br-int bridge with an additional node-local OVS port and an IP interface configuration (ohm-0) that depends on the definition (MAC, IP, MTU, etc.) in OpenStack.

The problem area

It appears that the current mechanism for setting the MTU on ohm-0 is probably only effective if the bridge "br-int" was not active beforehand.

If an operator uses an MTU of, for example, 9000 (a very common scenario) in the underlay/node network, this results in an automatically configured MTU of 8942 bytes for the Amphora load balancer network.

The Amphora VMs thus have also an automatically configured MTU of 8942 bytes on the interface to the Amphora management network. However, this is not the case on the ohm-0 interface, which the Health Manager uses for communication. Instead, an MTU of 1500 is used because the configuration command accepted but it is ignored.

This results in a problem if relatively large load balancers are now created (in our case, these consist of three listener ports and approximately 60 virtual machines on which the services are active), the Health Manager status packets grow to a size of more than 1500 bytes (~1750 bytes), as the Amphora Linux system assumes that the network can transport up to 8942 bytes. As a result, the packets no longer reach the Health Manager, which then continuously performs failovers of the Amphora VMs.

As an workaround, we reduced the MTU of the Amphora load balancer network to 1500 and performed failovers for all Amphorae. As a result, the error situation of constant failovers disappeared immediately, as the operating system automatically splits the payload into several packets when sending Health Manager status information.

Possible solutions to the problem

The configured MTUs of all communication partners and components involved must match each other. Ideally, the operator automates and checks this accordingly and completely.

Hint: For load balancers that interact with external networks that have an MTU greater than 1500, it must be ensured, at least in the case of UDP communication, that this is also the case outside the Openstack system (in the case of TCP, there is "Path MTU Discovery").

As a simple interim solution, it would make sense to make the MTU configuration mandatory for the Amphora load balancer network and limit it to a maximum of 1500 bytes for the typical case of underlays with jumbo frames.

Furthermore, the documentation should describe the consequences and background of the Amphora Management Network MTU configuration and its coupling to the MTU of the underlay network.