Add support for frr-k8s in Metallb Layer3 mode to be able to receive prefixes sent via BGP by external routes

When using Metallb in Layer3 mode, by using either native, frr or frr-k8s modes for BGP, the default setup implies that prefixes received from external routers are rejected. This means the received routes via BGP will not be added in the routing table of the nodes.

When load balancer services are exposed on additional node networks, populating the nodes routing table with the prefixes advertised by external routes is useful to assure symmetric traffic for the ingress flows. This can be achieved by using frr-k8s mode in which extra CR named FRRConfiguration can be used to adapt FRR configuration(which initially is fully configured by Metallb) via FRR-K8s api and be able to achieve the needed configuration.

Tested working configs that ensured end-to-end functionality for ingress on additional networks with frr-k8s are listed below. Cluster definition:

units:
  metallb:
    helmrelease_spec:
      values:
        frrk8s:
          enabled: true
cluster:
  metallb:
    bgp_lbs:
      l3_options:
        bgp_peers:
          - name: mx1
            local_asn: 64513
            peer_asn: 64512
            peer_address: 172.30.140.2
            passwordSecret: bgppeer-mx2-pass
            advertised_pools:
              - pool1
              - pool2
          - name: mx2
            local_asn: 64513
            peer_asn: 64512
            peer_address: 172.30.140.3
            passwordSecret: bgppeer-mx2-pass
            advertised_pools:
              - pool1
              - pool2
      address_pools:
        - name: pool1
          addresses:
            - 192.168.1.1-192.168.1.2
        - name: pool2
          addresses:
            - 192.168.2.1-192.168.2.2

FRRConfiguration manifests applied (when also BFD, BGP password, node selector are specified ):

  • receive all incoming prefixes, without filtering
---
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
 name: permit-route-learning-mx1
 namespace: metallb-system
spec:
  bgp:
    routers:
    - asn: 64513
      neighbors:
      - address: 172.30.140.2
        asn: 64512
        bfdProfile: bfd-profile-bgp
        passwordSecret:
          name: bgppeer-mx1-pass
        toReceive:
         allowed:
           mode: all
    bfdProfiles:
      - name: bfd-profile-bgp
        receiveInterval: 1000
        transmitInterval: 1000
        detectMultiplier: 3
  nodeSelector:
    matchExpressions:
     - key: node-role.kubernetes.io/control-plane
       operator: In
       values: ["true"]
---
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
 name: permit-route-learning-mx2
 namespace: metallb-system
spec:
  bgp:
    routers:
    - asn: 64513
      neighbors:
      - address: 172.30.140.3
        asn: 64512
        bfdProfile: bfd-profile-bgp
        passwordSecret:
          name: bgppeer-mx2-pass
        toReceive:
         allowed:
           mode: all
    bfdProfiles:
      - name: bfd-profile-bgp
        receiveInterval: 1000
        transmitInterval: 1000
        detectMultiplier: 3
  nodeSelector:
    matchExpressions:
     - key: node-role.kubernetes.io/control-plane
       operator: In
       values: ["true"]
  • only receive certian incoming prefixes, by using filtering
---
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
 name: permit-route-learning-mx1
 namespace: metallb-system
spec:
  bgp:
    routers:
    - asn: 64513
      neighbors:
       - address: 172.30.140.2
         asn: 64512
         bfdProfile: bfd-profile-bgp
         passwordSecret:
           name: bgppeer-mx1-pass
         toReceive:
           allowed:
             prefixes:
              - prefix: 2.2.2.0/30
              - prefix: 192.168.2.0/24
                ge: 25 
                le: 28
    bfdProfiles:
      - name: bfd-profile-bgp
        receiveInterval: 1000
        transmitInterval: 1000
        detectMultiplier: 3
  nodeSelector:
    matchExpressions:
     - key: node-role.kubernetes.io/control-plane
       operator: In
       values: ["true"]
---
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
 name: permit-route-learning-mx2
 namespace: metallb-system
spec:
  bgp:
    routers:
    - asn: 64513
      neighbors:
       - address: 172.30.140.3
         asn: 64512
         bfdProfile: bfd-profile-bgp
         passwordSecret:
           name: bgppeer-mx2-pass
         toReceive:
           allowed:
             prefixes:
              - prefix: 2.2.2.0/30
    bfdProfiles:
      - name: bfd-profile-bgp
        receiveInterval: 1000
        transmitInterval: 1000
        detectMultiplier: 3
  nodeSelector:
    matchExpressions:
     - key: node-role.kubernetes.io/control-plane
       operator: In
       values: ["true"]

Note1: Neighbors that do not need to learn routes do not need to be defined inside any FRRConfiguration object.

Note2: The best approach in FRRConfiguration CRs definition is to have one FRRConfiguration object per neighbor that needs to receive routes. This gives consistency to the configuration since the nodeSelector setting applies per metallb bgppeer, but inside FRRConfiguration is an general setting, not possible to define it per bgppeer.

Note3: All the extra setups like BFD, BGP passwords, node selector need to be specified in FRRConfiguration (although these are already part of bgppeers.metallb.io), otherwise frr config cannot be applied.

# kubectl apply -f metallb_frrk8s_extra_config.yaml
Error from server (Forbidden): error when creating "/root/net-team/manifests/metallb_frrk8s_extra_config.yaml": admission webhook "frrconfigurationsvalidationwebhook.metallb.io" denied the request: multiple bfd profiles specified for neighbor 172.30.140.2 at vrf
# k logs metallb-frr-k8s-bng8t -n metallb-system -c controller | grep "failed to apply the config"
{"caller":"frrconfiguration_controller.go:138","controller":"FRRConfigurationReconciler","error":"multiple passwords specified for neighbor 172.30.140.2 at vrf ","failed to apply the config":"metallb-system/permit-route-learning","level":"error","ts":"2025-03-05T09:28:09Z"}

Enabling route learning for metallb bgp_peers should be possible by using the values in the cluster definition. For example something like:

  • receive all incoming prefixes, without filtering
cluster:
  metallb:
    bgp_lbs:
      l3_options:
        bgp_peers:
          - name: mx1
            local_asn: 64513
            peer_asn: 64512
            peer_address: 172.30.140.2
            passwordSecret: bgppeer-mx1-pass
            advertised_pools:
              - pool1
              - pool2
            receive_routes:
              mode: all
          - name: mx2
            local_asn: 64513
            peer_asn: 64512
            peer_address: 172.30.140.3
            passwordSecret: bgppeer-mx2-pass
            advertised_pools:
              - pool1
              - pool2
            receive_routes:
              mode: all
  • only receive certian incoming prefixes, by using filtering
cluster:
  metallb:
    bgp_lbs:
      l3_options:
        bgp_peers:
          - name: mx1
            local_asn: 64513
            peer_asn: 64512
            peer_address: 172.30.140.2
            passwordSecret: bgppeer-mx1-pass
            advertised_pools:
              - pool1
              - pool2
            receive_routes:
              prefixes:
               - prefix: 2.2.2.0/30
               - prefix: 192.168.2.0/24
                 ge: 25 
                 le: 28 
          - name: mx2
            local_asn: 64513
            peer_asn: 64512
            peer_address: 172.30.140.3
            passwordSecret: bgppeer-mx2-pass
            advertised_pools:
              - pool1
              - pool2
            receive_routes:
              prefixes:
               - prefix: 2.2.2.0/30
Edited by Vlad Onutu