HelmReleases un-suspended during workload-cluster delete hooks

Summary

This issue has been observed in https://gitlab.com/sylva-projects/sylva-core/-/jobs/8488961692

Where delete-hook job failed on longhorn uninstallation, whereas the HelmRelease was supposed to be suspended to avoid the pruning of resources.

While looking at job logs, we cans see that HelmRelease has been suspended a the beginning of the job:

=== (PrepareForDeletion) / HelmRelease/kyverno / Suspending helmrelease
event/slyva-units.delete.sylva-units.28934fe2 created
=== (PrepareForDeletion) / HelmRelease/longhorn / Suspending helmrelease
event/slyva-units.delete.sylva-units.57195460 created
=== (PrepareForDeletion) / HelmRelease/longhorn-crd / Suspending helmrelease
event/slyva-units.delete.sylva-units.2b735c78 created
=== (PrepareForDeletion) / HelmRelease/monitoring / Suspending helmrelease
event/slyva-units.delete.sylva-units.345f207 created

But none of these HelmReleases have suspend: true flag set at the end of the job:

crustgather-job-8488961692 ~> k get hr -n kubeadm-capm3-virt -o yaml | yq '.items[].spec.suspend'
null
null
null
null
null
null
null
null
null
null

It turns out that periodic reconcilations of Kustomizations managing HelmReleases are removing that flag, this can easily be reproduced:

$ k patch hr cert-manager --type=merge --patch='{"spec":{"suspend":true}}'
helmrelease.helm.toolkit.fluxcd.io/cert-manager patched
$ 
$ k get hr cert-manager -o yaml | yq .spec.suspend
true
$ 
$ flux reconcile ks cert-manager
► annotating Kustomization cert-manager in sylva-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision sha1:a1300513df04168fdb9b292e9e50914860c2d3d4
$ 
$ k get hr cert-manager -o yaml | yq .spec.suspend
null