Calico / Tigera-operator tolerations leading to pods do be scheduled on draining node

Summary

Observed on https://gitlab.com/sylva-projects/sylva-core/-/jobs/7243615488 (upgrade of kubeadm management cluster)

We see that cp-2 machine fails to be deleted:

sylva-system         mgmt-1357074163-kubeadm-capm3-virt-control-plane-fjxgr   mgmt-1357074163-kubeadm-capm3-virt   mgmt-1357074163-kubeadm-capm3-virt-management-cp-0   metal3://sylva-system/mgmt-1357074163-kubeadm-capm3-virt-management-cp-0/mgmt-1357074163-kubeadm-capm3-virt-cp-eb3f4d18dc-zx4jp       Running        55m   v1.28.9
sylva-system         mgmt-1357074163-kubeadm-capm3-virt-control-plane-h7c6c   mgmt-1357074163-kubeadm-capm3-virt   mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   metal3://sylva-system/mgmt-1357074163-kubeadm-capm3-virt-management-cp-2/mgmt-1357074163-kubeadm-capm3-virt-cp-eb3f4d18dc-5dgcz       Deleting       95m   v1.28.9
sylva-system         mgmt-1357074163-kubeadm-capm3-virt-control-plane-mwcpb   mgmt-1357074163-kubeadm-capm3-virt   mgmt-1357074163-kubeadm-capm3-virt-management-cp-1   metal3://sylva-system/mgmt-1357074163-kubeadm-capm3-virt-management-cp-1/mgmt-1357074163-kubeadm-capm3-virt-cp-eb3f4d18dc-nfx9q       Running        95m   v1.28.9
sylva-system         mgmt-1357074163-kubeadm-capm3-virt-md0-ltd6j-xgw5p       mgmt-1357074163-kubeadm-capm3-virt                                                                                                                                                                                              Provisioning   39m   v1.28.9

CAPI keeps evicting tiger-operator pods

I0702 11:04:38.836420       1 machine_controller.go:910] "evicting pod tigera-operator/tigera-operator-795545875-njmzp\n" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="sylva-system/mgmt-1357074163-kubeadm-capm3-virt-control-plane-h7c6c" namespace="sylva-system" name="mgmt-1357074163-kubeadm-capm3-virt-control-plane-h7c6c" reconcileID="f124c124-e3d4-472c-8376-a3048d23ad39" KubeadmControlPlane="sylva-system/mgmt-1357074163-kubeadm-capm3-virt-control-plane" Cluster="sylva-system/mgmt-1357074163-kubeadm-capm3-virt" Node="mgmt-1357074163-kubeadm-capm3-virt-management-cp-2"
I0702 11:04:38.836473       1 machine_controller.go:910] "evicting pod tigera-operator/tigera-operator-795545875-mxxhq\n" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="sylva-system/mgmt-1357074163-kubeadm-capm3-virt-control-plane-h7c6c" namespace="sylva-system" name="mgmt-1357074163-kubeadm-capm3-virt-control-plane-h7c6c" reconcileID="f124c124-e3d4-472c-8376-a3048d23ad39" KubeadmControlPlane="sylva-system/mgmt-1357074163-kubeadm-capm3-virt-control-plane" Cluster="sylva-system/mgmt-1357074163-kubeadm-capm3-virt" Node="mgmt-1357074163-kubeadm-capm3-virt-management-cp-2"
I0702 11:04:38.836161       1 machine_controller.go:910] "evicting pod tigera-operator/tigera-operator-795545875-k9rxl\n" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="sylva-system/mgmt-1357074163-kubeadm-capm3-virt-control-plane-h7c6c" namespace="sylva-system" name="mgmt-1357074163-kubeadm-capm3-virt-control-plane-h7c6c" reconcileID="f124c124-e3d4-472c-8376-a3048d23ad39" KubeadmControlPlane="sylva-system/mgmt-1357074163-kubeadm-capm3-virt-control-plane" Cluster="sylva-system/mgmt-1357074163-kubeadm-capm3-virt" Node="mgmt-1357074163-kubeadm-capm3-virt-management-cp-2"
I0702 11:04:38.836667       1 machine_controller.go:910] "evicting pod tigera-operator/tigera-operator-795545875-ppq5l\n" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="sylva-system/mgmt-1357074163-kubeadm-capm3-virt-control-plane-h7c6c" namespace="sylva-system" name="mgmt-1357074163-kubeadm-capm3-virt-control-plane-h7c6c" reconcileID="f124c124-e3d4-472c-8376-a3048d23ad39" KubeadmControlPlane="sylva-system/mgmt-1357074163-kubeadm-capm3-virt-control-plane" Cluster="sylva-system/mgmt-1357074163-kubeadm-capm3-virt" Node="mgmt-1357074163-kubeadm-capm3-virt-management-cp-2"

Plenty of tigera-operator pods are scheduled on node-2

tigera-operator                     tigera-operator-795545875-hhtst                                              0/1     Terminating        0              9m36s   <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-hjfh7                                              0/1     Terminating        0              7m18s   <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-j2fvw                                              0/1     Terminating        0              92s     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-j4m9h                                              0/1     Terminating        0              13m     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-j887f                                              0/1     Terminating        0              27m     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-jzr2b                                              0/1     Terminating        0              29m     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-k27t7                                              0/1     Terminating        0              28m     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-k9rxl                                              0/1     Terminating        0              11m     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-kfr7v                                              0/1     Terminating        0              25m     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-kg7b9                                              0/1     Terminating        0              6m54s   <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-kghf6                                              0/1     Terminating        0              21m     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-khchw                                              0/1     Terminating        0              16m     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-ksq8j                                              0/1     Terminating        0              24m     <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2   <none>           <none>
tigera-operator                     tigera-operator-795545875-kv2mj                                              0/1     Terminating        0              115s    <none>           mgmt-1357074163-kubeadm-capm3-virt-management-cp-2

This seems to be related to https://github.com/projectcalico/calico/issues/6136

Edited Jul 02, 2024 by Francois Eleouet