MetalLB speaker pods can be evicted by Kubelet

I was playing with a dev workload cluster which was a bit short on disk space, and I observed the cluster becoming unreachable.

This was due to Kubelet having evicted the MetalLB speaker pods on the nodes:

root@my-capo-wc-cp-6508611af4-krc72:/home/node-admin# kubectl get pods -A -o wide |grep speaker
metallb-system        metallb-speaker-2wcdd                                     0/1     Evicted                       0               104s    <none>           my-capo-wc-cp-6508611af4-krc72   <none>           <none>
metallb-system        metallb-speaker-mbt88                                     0/1     Evicted                       0               104s    <none>           my-capo-wc-cp-6508611af4-x6572   <none>           <none>

Events:
  Type     Reason     Age    From               Message
  ----     ------     ----   ----               -------
  Normal   Scheduled  2m53s  default-scheduler  Successfully assigned metallb-system/metallb-speaker-2wcdd to my-capo-wc-cp-6508611af4-krc72
  Warning  Evicted    2m54s  kubelet            The node had condition: [DiskPressure].

This is a Kubelet eviction, against which the toleration set on the pods has no effect.

To avoid or mitigate this risk we should set a priorityClassName on these pods.

e.g for RKE2:

(metallb chart values)

speaker:
  priorityClassName: system-cluster-critical