Flux source-controller uses single-replica-storageclass sc even when longhorn is not the main CSI

Summary

While troubleshooting a failed job of !8661 (merged) I noticed that even thaw the default storageclass was rbd-csi, ceph controller is still using the longhorn single-replica-storageclass. The mechanism computing this should be fixed.

crustgather-job-15549343099 ~> kubectl get sc rbd-csi -o yaml | yq .metadata.annotations
meta.helm.sh/release-name: ceph-csi-rbd
meta.helm.sh/release-namespace: ceph-csi-rbd
storageclass.kubernetes.io/is-default-class: "true"
crustgather-job-15549343099 ~> kubectl get pvc -n flux-system flux-sources-pvc -o yaml |yq .spec
accessModes:
  - ReadWriteOnce
resources:
  requests:
    storage: 1Gi
storageClassName: single-replica-storageclass    <-------------
volumeMode: Filesystem
volumeName: pvc-6e513070-d360-4df2-bda1-6f77c0cc5770

Details