Enable monitoring and Grafana BGP dashboard for MetalLB frr-k8s mode

This issue involves implementing monitoring capabilities and adapting MetalLB Grafana dashboard to visualize BGP metrics for MetalLB when running with the FRR-K8s mode.

Configure Prometheus integration to scrape BGP-related metrics from frr-k8s pods instead of MetalLB speakers.

Example of values to enable frr-k8s:

metallb:
  bgp_lbs:
    l3_options:
      bgp_peers:
        ext-router1:
          local_asn: 64513
          peer_asn: 64513
          peer_address: 172.20.219.241
          advertised_pools:
            - pool1
          receive_routes:
            mode: all
    address_pools:
      pool1:
        addresses:
          - 192.168.1.1-192.168.1.2

Tasks:

(A) This config in metallb helm frr-k8s values helps in exposing frr-k8s metrics:

metallb_helm_values:
  frr-k8s:
    prometheus:
      namespace: cattle-monitoring-system
      rbacPrometheus: true
      serviceAccount: rancher-monitoring-prometheus
      serviceMonitor:
        enabled: true
      rbacProxy:
        repository: quay.io/brancz/kube-rbac-proxy
        tag: v0.12.0

(B) For Grafana metallb-dashboard configmap needs to be adapted because of different metric names:

Mode BGP Metric Name
Native BGP metallb_bgp_session_up
frr-k8s frrk8s_bgp_session_up

In frr-k8s mode, the frr-metrics container exports metrics with the frrk8s_bgp_* prefix. The Grafana dashboard queries metallb_bgp_*.

(C) For Thanos, sylva-thanos-rules-configmap namespace contains existing MetalLB BGP alert rules in its metallb.yml key:

  • MetalLB-BGP_Session_Down — fires when metallb_bgp_session_up == 0
  • MetalLB-BGP_All_Sessions_Down — fires when all metallb_bgp_session_up are 0

Both query metallb_bgp_session_up, the alerts will never fire for any cluster running in frr-k8s mode. Patch the metallb.yml key in sylva-thanos-rules-configmap to extend each BGP alert expression with an or covering frrk8s_bgp_session_up.

(D) Adapt also how values are passed to sylva-capi-cluster for frr-k8s mode for first node, to omit serviceMonitor enablement similar to what is done for MetalLB: https://gitlab.com/sylva-projects/sylva-core/-/blob/1.6.18/charts/sylva-units/values.yaml?ref_type=tags#L10652

To avoid errors like:

wc1-vlado-cp-081544d333-c5fbd:~ # kubectl -n metallb-system logs helm-install-metallb-6875q
...
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "metallb-frr-k8s-frr-k8s-monitor" namespace: "metallb-system" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first
Edited by Vlad Onutu