workload cluster ingress-nginx healthcheck done in mgmt cluster

(reported by Nabil Aïssiou)

[in] the ingress-nginx unit level (in the case of a workload cluster) [...] there is a healthcheck at the ingress-nginx Kustomization level:

healthChecks:

• apiVersion: apps/v1 kind: DaemonSet name: ingress-nginx-controller # on sylva upstream: rke2-ingress-nginx-controller namespace: ingress-nginx # on sylva upstream: kube-system

The healthcheck verifies the nginx DaemonSet deployed on the management cluster and not the one on the workload cluster. If nginx daemonset is not working on the management cluster, it blocks reconciliation at the workload cluster level

Reproduction procedure:

• Create a malfunction on the management cluster's nginx (example: use a non-existent nginx Docker image)
• Attempt to execute an apply-workload
• Observe the reconciliation blockage

the underlying problem is something that we solved on some other units:

• the HelmRelease resource does not have some healthcheck ability that a Kustomization has
• so we check the resource by adding a healthCheck in the Kustomization
• this can't work in the case where the unit deployed in the workload cluster, because in that case the Kustomization, which produces the HelmRelease in the mgmt cluster, has the mgmt cluster as the target namespace ... so we end-up checking things in the mgmt cluster
• the solution in that case is to split the unit in to two: here we would split the ingress-nginx unit into an ingress-nginx having no kustomization_spec.healthChecks and an ingress-nginx-ready unit that will have the healthCheck  (and units that depend on nginx may be updated to depend on nginx-ingress-ready, if this is better adapted)