add Kyverno policy to prevent recreation of RKE2 HelmCharts for Calico and MetalLB

Thomas Morinrequested to merge
prevent-rke2-helmcharts-calico-metallb into main

This MR introduces a policy to prevents the creation of RKE2 HelmChart.helm.cattle.io resources for Calico and MetalLB.

The reason is that with !3101 (merged) and !3218 (merged), Calico and MetalLB are managed by FluxCD not by RKE2.

We need to prevent the recreation of RKE2 HelmCharts for those to ensure that when we transition from the old way (when those were managed by RKE2) and the new way, we don't have old RKE2 nodes recreating these HelmChart resources.

This is brought as a separate MR because this is more a safeguard, we don't believe that this is likely to occur (and hence does not need to be included in !3101 (merged) and !3218 (merged)).

With this MR with our original code where RKE2 manages metallb, I have this when deleting the HelmChart and restarting the rke2-server unit on a CP node:

Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]: I1104 15:41:45.650224  212130 event.go:376] "Event occurred" object="kube-system/metallb" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Warning" reason>
Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]:         Applying manifest at "/var/lib/rancher/rke2/server/manifests/metallb.yaml" failed: failed to create metallb-system/metallb helm.cattle.io/v1, Kind=HelmChart fo>
Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]:         
Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]:         resource HelmChart/metallb-system/metallb was blocked due to the following policies
Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]:         
Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]:         prevent-rke2-helmcharts-calico-metallb:
Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]:           prevent-metallb-calico: With Sylva, Calico and MetalLB are managed by FluxCD not
Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]:             by RKE2, the creation of RKE2 HelmCharts for those is hence blocked.
Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]:         , the server could not find the requested resource, the server could not find the requested resource
Nov 04 15:41:45 management-cluster-cp-27e688539c-7m2pg rke2[212130]:  >

/cc @cristian.manda

Edited by Thomas Morin

Merge request reports