auto-provision internal Neutron network for Octavia Amphora Management instead of provider network

Supersedes: !2791 (closed)

Depends-On: yaook/images/neutron-ovn-agent!108 (merged)

Depends-On: !2934 (merged)

This rebuilds the approach taken for connecting Amphora VMs with the Octavia services (health manager, worker). The goal is to remove the necessity of using a provider network and instead create an internal regular Neutron network for which local OVS internal port interfaces are plugged on each node that runs the Octavia services to connect them to the network. This aligns with the approach that OpenStack Kolla uses to integrate Octavia.

Note: please see the "YAOOK integration architecture" appendix section of the "Octavia operator" handbook chapter as updated by this MR. It contains explanations and figures of the new integration approach. Here is an intermediate build. The main points are:

Neutron and OVS port handling:

• A new OctaviaNeutronPort CR is introduced. This is necessary to dynamically manage per-node OpenStack Neutron ports which are used to connect some of the Octavia services (health managers and workers) with the Amphora VM network in OpenStack.
  • The octavia-operator is both producer and consumer of these resources: in the OctaviaDeployment reconcile loop it creates these resources and waits for their reconcile to retrieve the resulting port data (MAC/IP address etc.) whereas in a dedicated OctaviaNeutronPort reconcile loop, the octavia-operator takes care of materializing these resources in OpenStack.
• For each Kubernetes node that will host the Octavia services which need access to the Amphora VMs (health manager, worker), a node-specific Neutron Port in the Neutron network for the Amphora VMs is created, which will be materialized as an OVS internal port on the integration bridge on each of those nodes.
• Since the Octavia worker also requires access to the OVS port in order to connect to the Amphora VMs, it shares the same scheduling node label with the health manager now and simply reuses the node-local OVS port plugged by the health manager.
• The Octavia health manager (HM) Pod incorporates an init container which plugs a local OVS internal port on the integration bridge of the Kubernetes node which materializes the node-specific Neutron port by assigning its MAC and port ID.
• In order to be able to use the OVS internal port, all Octavia service Pods relying on it are run in host network mode.
• OVS ports keep existing even after health managers are removed from a node. Since their Neutron Port counterparts are deleted on the other hand, their flow rules are removed and they become inactive.

Static resource handling:

• Octavia will create some static OpenStack resources in the "service" project that it will use create the Neutron ports:
  • one Neutron network with subnet to connect Amphora VMs, Octavia health managers and Octavia workers
  • two Security Groups: one for the Amphora VMs and one for the health managers
• Since the operator cannot prevent resources from being attached to these networks or security groups within OpenStack outside of Octavia, they might fail to delete (because they are still in use) when the OctaviaDeployment CR is removed. In such case, the operator will rename them and append an "-orphaned" suffix to avoid them getting reused in the future (since leftover mismatching CIDRs may collide etc.).

Other notable changes:

• The octavia-operator will now create two KeystoneUsers:
  • octavia: (existing user) used by the reconcile loop for OctaviaDeployment, manages the static resources like network, security groups and so on
  • octavia-resources: (new) used by the reconcile loop for OctaviaNeutronPort, manages ports in Neutron
• All Octavia services that require access to the Amphora network share the same scheduling key as the Octavia health manager because they require the OVS internal port that is plugged by it and thus need to reside on the same Kubernetes node.
  • Octavia worker and health manager now use a new combined scheduling key called octavia.yaook.cloud/octavia-managers.
  • This Octavia scheduling key is added to yaook.op.scheduling_keys.OVN_SCHEDULING_KEYS to ensure that an OVN controller is present on each node the Octavia worker and health manager services are, since Octavia has to use ovs-vsctl to plug the OVS port on the integration bridge in order to access the Amphora Neutron network.

Note when migrating from old setups:

• The octaviaConfig.health_manager.controller_ip_port_list entry must be removed from OctaviaDeployment spec!
• The octaviaConfig.controller_worker.amp_boot_network_list entry must be removed from OctaviaDeployment spec!
• The octaviaConfig.controller_worker.amp_secgroup_list entry must be removed from OctaviaDeployment spec!
• Remove replicas: 1 from the health_manager and worker sections of the OctaviaDeployment spec, set them to {} if there are no other entries.
• Add networking.managementSubnetCIDR to the OctaviaDeployment spec.
• Delete the Kubernetes Service belonging to the health manager.