pivot fails because deletion is forbidden by kyverno
Summary
Seen on local capm3 libvirt. Pivot job breaks because of kyverno
$ k logs -n kube-job pivot-job-sylva-system-nb7hj
-- Signal that the pivot job has started. This is used in bootstrap.sh to prevent accidental re-runs
kustomization.kustomize.toolkit.fluxcd.io/cluster annotated
secret/management-cluster-kubeconfig-copy created
-- Retrieve target cluster kubeconfig
-- Wait for cluster and machines to be ready as it is a required condition to move
cluster.cluster.x-k8s.io/management-cluster condition met
machine.cluster.x-k8s.io/management-cluster-control-plane-9k2fp condition met
-- Wait for all Kustomizations related to Cluster API to be ready in management cluster
kustomization.kustomize.toolkit.fluxcd.io/capi-providers-pivot-ready condition met
-- Suspend Kustomizations and HelmReleases in bootstrap cluster that relate to the management cluster
helmrelease.helm.toolkit.fluxcd.io/cluster patched
helmrelease.helm.toolkit.fluxcd.io/metal3-suse patched
kustomization.kustomize.toolkit.fluxcd.io/cluster patched
kustomization.kustomize.toolkit.fluxcd.io/management-cluster-flux patched
kustomization.kustomize.toolkit.fluxcd.io/management-sylva-units patched
kustomization.kustomize.toolkit.fluxcd.io/metal3-suse patched
kustomization.kustomize.toolkit.fluxcd.io/namespace-defs patched
helmrelease.helm.toolkit.fluxcd.io/sylva-units patched
-- Move cluster definitions from source to target cluster
Installing the clusterctl inventory CRD
Installing the clusterctl inventory CRD
Performing move...
Discovering Cluster API objects
Total objects Count=52
Moving Cluster API objects Clusters=1
Moving Cluster API objects ClusterClasses=0
Pausing the source cluster
Pausing the source ClusterClasses
Waiting for all resources to be ready to move
Creating objects in the target cluster
Creating Cluster="management-cluster" Namespace="sylva-system"
Creating IPPool="management-cluster-primary-pool" Namespace="sylva-system"
Creating Metal3DataTemplate="management-cluster-cp-metadata-c00fb1c81a" Namespace="sylva-system"
Creating ConfigMap="management-cluster-lock" Namespace="sylva-system"
Creating IPPool="management-cluster-provisioning-pool" Namespace="sylva-system"
Creating RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Creating Secret="management-cluster-token" Namespace="sylva-system"
Creating Metal3Cluster="management-cluster" Namespace="sylva-system"
Creating Secret="management-cluster-cca" Namespace="sylva-system"
Creating Machine="management-cluster-control-plane-9k2fp" Namespace="sylva-system"
Creating Secret="management-cluster-kubeconfig" Namespace="sylva-system"
Creating Secret="management-cluster-ca" Namespace="sylva-system"
Creating Metal3Machine="management-cluster-cp-fbdf064963-w9spd" Namespace="sylva-system"
Creating RKE2Config="management-cluster-control-plane-xt5cs" Namespace="sylva-system"
Creating Secret="management-cluster-control-plane-xt5cs" Namespace="sylva-system"
Creating BareMetalHost="management-cluster-management-cp-0" Namespace="sylva-system"
Creating Metal3DataClaim="management-cluster-cp-fbdf064963-w9spd" Namespace="sylva-system"
Creating HostFirmwareSettings="management-cluster-management-cp-0" Namespace="sylva-system"
Creating Metal3Data="management-cluster-cp-metadata-c00fb1c81a-0" Namespace="sylva-system"
Creating HardwareData="management-cluster-management-cp-0" Namespace="sylva-system"
Creating Secret="management-cluster-management-cp-0-secret" Namespace="sylva-system"
Creating Secret="management-cluster-cp-fbdf064963-w9spd-metadata" Namespace="sylva-system"
Creating Secret="management-cluster-cp-fbdf064963-w9spd-networkdata" Namespace="sylva-system"
Creating IPClaim="management-cluster-management-cp-0-management-cluster-provisioning-pool" Namespace="sylva-system"
Creating IPClaim="management-cluster-management-cp-0-management-cluster-primary-pool" Namespace="sylva-system"
Creating FirmwareSchema="schema-f229959d" Namespace="sylva-system"
Creating IPAddress="management-cluster-prov-192-168-10-20" Namespace="sylva-system"
Creating IPAddress="management-cluster-bmv4-192-168-100-20" Namespace="sylva-system"
Deleting objects from the source cluster
Deleting IPAddress="management-cluster-prov-192-168-10-20" Namespace="sylva-system"
Deleting IPAddress="management-cluster-bmv4-192-168-100-20" Namespace="sylva-system"
Deleting Secret="management-cluster-cp-fbdf064963-w9spd-metadata" Namespace="sylva-system"
Deleting Secret="management-cluster-cp-fbdf064963-w9spd-networkdata" Namespace="sylva-system"
Deleting IPClaim="management-cluster-management-cp-0-management-cluster-provisioning-pool" Namespace="sylva-system"
Deleting IPClaim="management-cluster-management-cp-0-management-cluster-primary-pool" Namespace="sylva-system"
Deleting FirmwareSchema="schema-f229959d" Namespace="sylva-system"
Deleting HostFirmwareSettings="management-cluster-management-cp-0" Namespace="sylva-system"
Deleting Metal3Data="management-cluster-cp-metadata-c00fb1c81a-0" Namespace="sylva-system"
Deleting HardwareData="management-cluster-management-cp-0" Namespace="sylva-system"
Deleting Secret="management-cluster-management-cp-0-secret" Namespace="sylva-system"
Deleting Secret="management-cluster-control-plane-xt5cs" Namespace="sylva-system"
Deleting BareMetalHost="management-cluster-management-cp-0" Namespace="sylva-system"
Deleting Metal3DataClaim="management-cluster-cp-fbdf064963-w9spd" Namespace="sylva-system"
Deleting Metal3Machine="management-cluster-cp-fbdf064963-w9spd" Namespace="sylva-system"
Deleting RKE2Config="management-cluster-control-plane-xt5cs" Namespace="sylva-system"
Deleting Secret="management-cluster-cca" Namespace="sylva-system"
Deleting Machine="management-cluster-control-plane-9k2fp" Namespace="sylva-system"
Deleting Secret="management-cluster-kubeconfig" Namespace="sylva-system"
Deleting Secret="management-cluster-ca" Namespace="sylva-system"
Deleting IPPool="management-cluster-primary-pool" Namespace="sylva-system"
Deleting Metal3DataTemplate="management-cluster-cp-metadata-c00fb1c81a" Namespace="sylva-system"
Deleting ConfigMap="management-cluster-lock" Namespace="sylva-system"
Deleting IPPool="management-cluster-provisioning-pool" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting RKE2ControlPlane="management-cluster-control-plane" Namespace="sylva-system"
Deleting Secret="management-cluster-token" Namespace="sylva-system"
Deleting Metal3Cluster="management-cluster" Namespace="sylva-system"
Error: action failed after 10 attempts: error deleting "controlplane.cluster.x-k8s.io/v1alpha1, Kind=RKE2ControlPlane" sylva-system/management-cluster-control-plane: admission webhook "validate.kyverno.svc-fail" denied the request:
resource RKE2ControlPlane/sylva-system/management-cluster-control-plane was blocked due to the following policies
avoid-delete-mgmt-resources:
cluster-resources: Deleting the Sylva Flux resources that describe the Cluster API
cluster for the management cluster is not possible without breaking the Sylva
management cluster.Other pivot tries also fail because kubeconfig secret was deleted
$ k logs -n kube-job pivot-job-sylva-system-s6jnp
-- Signal that the pivot job has started. This is used in bootstrap.sh to prevent accidental re-runs
kustomization.kustomize.toolkit.fluxcd.io/cluster annotated
Error from server (NotFound): secrets "management-cluster-kubeconfig" not found
error: no objects passed to applyEdited by Médéric de Verdilhac