Make Minio optional on management cluster if external S3 bucket is available (for Thanos, Loki and potentially backup)

Summary

Both Loki and Thanos need a S3 bucket as storage backend, the current solution is deploying a Minio cluster on top of cluster storage backend (e.g. Longhorn or Cinder). However, on many use-cases, we could avoid this Minio Unit and storage consumption by reusing existing Object storage solution (e.g. through Ceph RadosGW) which can be used and shared also for backup (see: Draft: Add unit to backup CAPI resources of any clusters (!3813) · Merge requests · Sylva-projects / sylva-core · GitLab). The Management cluster definition should provide option to reuse an existing S3 bucket and not oblige Minio installation.

Details

Add the possibility to deactivate Minio units and defined external s3 configuration (defined in the management cluter values.yaml ) to inject it in thanos.helmrelease_spec.objstoreConfig: and loki.helmrelease_spec and potentially in future backup unit

units  
  minio-logging
    enabled: false
  minio-monitoring
    enabled: false
s3:
  shared_s3: #shared s3 configuration
    enabled: true
    host: <s3-host>
    accessKey: <ak>
    secretKey: <sk>
    cert: <s3-host certificate if relevant>
  
  loki_s3: #specific unit s3 configuration
    bucket: <bucket name>

  thanos_s3:
    bucket: <bucket name>

#  backup_s3:
#    enabled: true
#    host: <s3-host>
#    accessKey: <ak>
#    secretKey: <sk>
#    bucket: <bucket name>
#    cert: <s3-host certificate if relevant>