longhorn-instance-manager-cleanup job failing in workload clusters
Summary
When longhorn is enabled in workload clusters, longhorn-instance-manager-cleanup jobs are failing with this error:
ubuntu@bootstrap-nokia-cfx:~/my-deployment/sylva-core$ kubectl logs longhorn-instance-manager-cleanup-28799160-6gn77 -n sylva-system --kubeconfig nokia-cfx-kubeconfig
=== Checking for nodes being drained ===
=== Node nokia-cfx-my-hp-server-3 is being drained, checking if it has volumes attached ===
Error from server (Forbidden): volumes.longhorn.io is forbidden: User "system:serviceaccount:sylva-system:longhorn-instance-manager-cleanup-sa" cannot list resource "volumes" in API group "longhorn.io" in the namespace "longhorn-system"
This job is supposed to run in longhorn-system namespace, in which it has permssions to list volumes. This jobs runs in longhorn-system in management cluster, but that's apparently not the case for workload clusters as we force kustomizations to taget sylva-system namespace here.
This can be fixed by setting th right target in workload-cluster.values.yaml:
units:
longhorn-instance-manager-cleanup:
kustomization_spec:
targetNamespace: longhorn-system