workload-cluster deletion blocked because HelmRelease in bad state

I ran into the following case in my dev env:

  • workload cluster in bad state (Calico had been removed)
  • impossible to have the kyverno HelmRelease reconcile => this HelmRelease is stuck
  • on workload cluster deletion (kubectl delete -n wc-xxx hr sylva-units) the Helm pre-delete hook is stuck, because the kyverno HelmRelease can't be deleted:
...
=== Remaining kustomizations ===
cluster-reachable kyverno mgmt-cluster-ready namespace-defs root-dependency-22
=== (Deletion) / Kustomization/kyverno / Has to delete
event/slyva-units.delete.sylva-units.65e07feb created
Remaining resource: HelmRelease my-capo-wc/kyverno
=== Remaining kustomizations ===
cluster-reachable kyverno mgmt-cluster-ready namespace-defs root-dependency-22
=== (Deletion) / Kustomization/kyverno / Has to delete
event/slyva-units.delete.sylva-units.3b894fb2 created
Remaining resource: HelmRelease my-capo-wc/kyverno
=== Remaining kustomizations ===
cluster-reachable kyverno mgmt-cluster-ready namespace-defs root-dependency-22
=== (Deletion) / Kustomization/kyverno / Has to delete
event/slyva-units.delete.sylva-units.1f361bb3 created
Remaining resource: HelmRelease my-capo-wc/kyverno
=== Remaining kustomizations ===
cluster-reachable kyverno mgmt-cluster-ready namespace-defs root-dependency-22
=== (Deletion) / Kustomization/kyverno / Has to delete
event/slyva-units.delete.sylva-units.5c1672df created
Remaining resource: HelmRelease my-capo-wc/kyverno
=== Remaining kustomizations ===
cluster-reachable kyverno mgmt-cluster-ready namespace-defs root-dependency-22
=== (Deletion) / Kustomization/kyverno / Has to delete
event/slyva-units.delete.sylva-units.74b75d7e created
Remaining resource: HelmRelease my-capo-wc/kyverno
=== Remaining kustomizations ===
cluster-reachable kyverno mgmt-cluster-ready namespace-defs root-dependency-22
=== (Deletion) / Kustomization/kyverno / Has to delete
event/slyva-units.delete.sylva-units.7bb363c9 created
Remaining resource: HelmRelease my-capo-wc/kyverno
=== Remaining kustomizations ===
cluster-reachable kyverno mgmt-cluster-ready namespace-defs root-dependency-22
=== (Deletion) / Kustomization/kyverno / Has to delete
event/slyva-units.delete.sylva-units.3bee1acd created
Remaining resource: HelmRelease my-capo-wc/kyverno
=== Remaining kustomizations ===
cluster-reachable kyverno mgmt-cluster-ready namespace-defs root-dependency-22
=== (Deletion) / Kustomization/kyverno / Has to delete
event/slyva-units.delete.sylva-units.1c2b42bc created

I think that in sylva-units Helm pre-delete hook we should perhaps patch the HelmRelease that have a kubeConfig set to deploy in the workload cluster, to not run any hook and not require any actual deletion of remote resources (spec.uninstall:, deletionPropagation=orphan + disableHooks: true disableWait: true timeout: something short), or even patch them to remove FluxCd finalizer

/cc @feleouet

Assignee Loading
Time tracking Loading