Refactor metallb-resources helm chart to enhance BGP features

Summary

Current metallb-resources Helm values schema is limiting advanced BGP MetalLB functionalities for BGPAdvertisement, because a single BGPAdvertisement object is created per BGP peer. Although this is a working setup in many topologies, being able to support more MetalLB BGP features in cluster definition requires a redesign of the metallb-resources helm chart.

Also, take the opportunity to introduce the ability to specify vrf and ebgp_multihop for MetalLB BGP peers, for experimental use.

Proposed changes

Targeted new features for BGPAdvertisement

root@vlado-jump:~/git_folder/sylva-core# k explain bgpadvertisements.metallb.io.spec
GROUP:      metallb.io
KIND:       BGPAdvertisement
VERSION:    v1beta1

FIELD: spec <Object>


DESCRIPTION:
    BGPAdvertisementSpec defines the desired state of BGPAdvertisement.

FIELDS:
  aggregationLength     <integer>
    The aggregation-length advertisement option lets you “roll up” the /32s into
    a larger prefix. Defaults to 32. Works for IPv4 addresses.

  communities   <[]string>
    The BGP communities to be associated with the announcement. Each item can be
    a standard community of the
    form 1234:1234, a large community of the form large:1234:1234:1234 or the
    name of an alias defined in the
    Community CRD.

  localPref     <integer>
    The BGP LOCAL_PREF attribute which is used by BGP best path algorithm,
    Path with higher localpref is preferred over one with lower localpref.

  nodeSelectors <[]Object>
    NodeSelectors allows to limit the nodes to announce as next hops for the
    LoadBalancer IP. When empty, all the nodes having  are announced as next
    hops.

  peers <[]string>
    Peers limits the bgppeer to advertise the ips of the selected pools to.
    When empty, the loadbalancer IP is announced to all the BGPPeers configured.

Furthermore, named communities should be supported:

apiVersion: metallb.io/v1beta1
kind: Community
metadata:
  name: communities
  namespace: metallb-system
spec:
  communities:
  - name: vpn-only
    value: 1234:1
  - name: NO_ADVERTISE
    value: 65535:65282

And named communities could be used inside BGPAdvertisements:

apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: local
  namespace: metallb-system
spec:
  ipAddressPools:
  - first-pool
  communities:
  - vpn-only

Proposed metallb-resources values schema change under metallb.bgp_lbs.l3_options:

Item Change Observation
update bgp_peers remove advertised_pools under <bgp_peer_name> bgp peers will be mentioned under bgp_advertisements.<bgp_adv_name>.peers
add bgp_advertisements define named BGPAdvertisements see next table
add communities collection of aliases for communities see example above

Proposed metallb-resources values schema change under metallb.bgp_lbs.l3_options.bgp_advertisement.<bgp_advertisement_name>:

Item Change Observation
aggregationLength optional The aggregation-length advertisement option lets you “roll up” the /32s into a larger prefix. Defaults to 32
localPref optional The BGP LOCAL_PREF attribute which is used by BGP best path algorithm, path with higher localpref is preferred over one with lower localpref.
nodeSelectors optional NodeSelectors allows to limit the nodes to announce as next hops for the LoadBalancer IP. When empty, all the nodes having are announced as next hops.
peers mandatory Peers limits the bgppeer to advertise the ips of the selected pools to. When empty, the loadbalancer IP is announced to all the BGPPeers configured.
communities optional The BGP communities to be associated with the announcement. Each item can be a standard community of the form 1234:1234, a large community of the form large:4534:6233:1234 or the name of an alias defined in the Community CRD.
advertised_pools mandatory The list of IPAddressPools to advertise via this advertisement, selected by name

Example of new Metallb BGP definition, comments are showing the proposed evolution:

metallb:
  bgp_lbs:
    address_pools:
      pool1:
        addresses:
          - 192.168.1.1-192.168.1.2
      pool2:
        addresses:
          - 192.168.2.1-192.168.2.2
    l3_options:
      bfd_profiles:
        bfd-profile-bgp:
          receiveInterval: 1000
          transmitInterval: 1000
          detectMultiplier: 3
      bgp_peers:
        mx1:
          local_asn: 64513
          peer_asn: 64512
          peer_address: 172.30.140.2
          password: bgp-peer-password
          bfd_profile: bfd-profile-bgp
          node_selectors: # should be adapted to changed for consistency with bgp_advertisements
            - matchLabels:
                kubernetes.io/os: linux   
          # advertised_pools: # removed in this proposed change
          #   - pool1         # removed in this proposed change  
          #   - pool2         # removed in this proposed change   
          receive_routes:
            mode: all
        mx2:
          local_asn: 64513
          peer_asn: 64512
          peer_address: 172.30.140.3
          password: bgp-peer-password
          bfd_profile: bfd-profile-bgp
          node_selectors: # should be adapted to changed for consistency with bgp_advertisements
            - matchLabels:
                kubernetes.io/os: linux     
          # advertised_pools: # removed in this proposed change
          #   - pool1         # removed in this proposed change  
          #   - pool2         # removed in this proposed change               
          receive_routes:
            prefixes:
             - prefix: 10.200.200.1/32
             - prefix: 2.2.2.0/24
               ge: 30
               le: 30
      communities:  # new - optional
       - name: vpn1-comm
         value: 1234:1
       - name: vpn2-comm
         value: 1234:2
      bgp_advertisements:  # new - mandatory
        bgp-adv1:  # new - mandatory
          advertised_pools:  # new - mandatory
            - pool1         
          peers:  # new - mandatory
            - mx1
            - mx2
          aggregationLength: 32  # new - optional
          communities:  # new - optional
            - 64512:100
            - vpn1-comm
          localPref: 200  # new - optional
          nodeSelectors:  # new - optional
            - matchLabels:
                kubernetes.io/os: linux
        bgp-adv2:  # new - mandatory
          advertised_pools:  # new - mandatory
            - pool2       
          peers:  # new - mandatory
            - mx1
            - mx2
          aggregationLength: 32  # new - optional
          communities:  # new - optional
            - 64512:200
            - vpn2-comm
          localPref: 400  # new - optional
          nodeSelectors:  # new - optional
            - matchLabels:
                kubernetes.io/os: linux

Targeted new features for BGPPeer

  • Introduce the ability to specify vrf for MetalLB BGPPeers, for experimental use:
$ k explain bgppeers.metallb.io.spec.vrf
GROUP:      metallb.io
KIND:       BGPPeer
VERSION:    v1beta2

FIELD: vrf <string>


DESCRIPTION:
    To set if we want to peer with the BGPPeer using an interface belonging to
    a host vrf

Note: could be used in the future for symmetric routing when exposing LB services on secondary networks similar to: https://docs.okd.io/4.19/networking/ingress_load_balancing/metallb/metallb-configure-return-traffic.html#nw-metallb-configure-return-traffic-proc_metallb-configure-return-traffic

  • Introduce the ability to define BGP peerings with multi-hop enabled for external BGP.
$ k explain bgppeers.metallb.io.spec.ebgpMultiHop
GROUP:      metallb.io
KIND:       BGPPeer
VERSION:    v1beta2

FIELD: ebgpMultiHop <boolean>


DESCRIPTION:
    To set if the BGPPeer is multi-hops away. Needed for FRR mode only.

Prepare and test migration for production

We need to implement the change in a manner which will avoid or reduce to minimum the LB services impact on running clusters. A solution could be to not delete the legacy bgp_advertisements and clean them manually after cluster is upgraded to the new metallb-resources Helm chart version.

Edited by Vlad Onutu