Release-1.6: Nightly pipeline fails due to missing kubelet RBAC permission

The nightly test pipelines for release-1.6 are failing on update-workload-cluster. When the workload cluster gets upgraded to a newer K8s version, Calico also gets upgraded from version 3.31 to 3.32 at the same time. This breaks the cluster as nodes lose etcd connectivity and Calico ends up in a Failed state.

A recent change !7861 (merged) split the Calico version rule so that Kubernetes 1.33+ now pulls Calico 3.32 instead of 3.31 and calico 3.32 has some breaking changes.

Failure explanation:

  • The failure happens during the Kubernetes upgrade from 1.32.13 to 1.33.13. It is not caused by the Calico upgrade.
  • Kubernetes 1.33.13 introduces a security change from https://github.com/kubernetes/kubernetes/pull/138957 that requires a new ClusterRoleBinding for the API server to communicate with kubelets.
  • Release-1.6 uses Cluster API v1.10.9, which does not create this binding when upgrading an existing cluster to k8s 1.33.13
  • Without the binding, Cluster API cannot check etcd health, reports EtcdMemberHealthy: Unknown, and blocks the upgrade.
  • The release-1.6 fix is to create the required ClusterRoleBinding in the workload cluster before starting the Kubernetes upgrade. Release-1.7 works because Cluster API v1.12.8 already includes this fix.

The MR pipeline that introduced this change did not test upgrade from the old version. So, the bug was not caught.

Failure pipelines:

Edited by Priya Goyal