MetalLB frr-k8s: minimize LB service downtime on upgrades from 1.6.x to 1.7.y
Problem
Clusters running MetalLB in native BGP mode (no route learning or BFD enabled) that are upgraded from 1.6.x to 1.7.y experience a period of total LB service outage during the upgrade.
The root cause is that 1.7.0 introduces frr-k8s as the default BGP backend for all clusters using L3 options, replacing the native BGP mode used in earlier releases (when ugraded clusters are not having route learning or BFD enabled), meaning (1) -> (3) from the table below.
| case | Sylva release |
route learning or BFD enabled on BGP Peers? |
BGP backend |
|---|---|---|---|
| (1) | 1.6.x | no | native BGP |
| (2) | 1.6.x | yes | frr-k8s |
| (3) | 1.7.y | no | frr-k8s |
| (4) | 1.7.y | yes | frr-k8s |
This means that during the MetalLB Helm upgrade, speaker pods are restarted with a new image version while a new frr-k8s DaemonSet is deployed for the first time. The combination of image pulls across all nodes and the BGP session re-establishment causes the downtime window.
The chart applies the frr-k8s DaemonSet and the speaker rolling update simultaneously with no ordering between them. Nodes could end up with the new speakers running but no BGP session if frr-k8s is still pulling its images.
The outage is a race between Metallb speaker rolling update completion speed and frr-k8s image pull and readiness.
Clusters using MetalLB in L2 mode only are not affected.
Requirement
Add a sylva-core unit that pre-pulls the new container images on all nodes before the MetalLB upgrade reconciles, so that the speaker restart and the first-time frr-k8s DaemonSet scheduling are not delayed by image pulls.
The unit must:
- be active only during upgrades and only on clusters with BGP configured
- have no effect on fresh installs or L2-only clusters
- complete before the MetalLB unit starts to reconcile
Images to pre-pull - 1.7.0 snapshot
The following images need to be present on all nodes before the upgrade proceeds:
| Image | Notes |
|---|---|
quay.io/metallb/frr-k8s:v0.0.25 |
new in main if native BGP mode was used; version bump from v0.0.21 if BFD or route learning was already enabled in release-1.6 |
quay.io/frrouting/frr:10.4.3 |
new in main if native BGP mode was used; version bump from 10.4.1 if BFD or route learning was already enabled in release-1.6 |