minio cert generation issues

Summary

We've observed various deployment where minio-tenant creation or update was stopped because there were errors on certificate generation:

Error on the CSR:

I0228 00:06:36.568912       1 minio.go:308] Generating private key
I0228 00:06:36.569007       1 minio.go:321] Generating CSR with CN=*.logging-hl.minio-logging.svc.cluster.local
I0228 00:06:36.592056       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0228 00:06:36.601620       1 csr.go:181] Start polling for certificate of csr/logging-minio-logging-csr, every 5s, timeout after 20m0s
I0228 00:06:36.601656       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-logging", Name:"logging", UID:"f6d79f13-f17f-48ca-be7e-1a3131876fa9", APIVersion:"minio.min.io/v2", ResourceVersion:"917958", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' MinIO CSR Created
I0228 00:06:38.884260       1 status.go:89] Hit conflict issue, getting latest version of tenant
E0228 00:06:41.604683       1 csr.go:203] Unexpected error during certificate fetching of csr/logging-minio-logging-csr V1: certificatesigningrequests.certificates.k8s.io "logging-minio-logging-csr" not found
E0228 00:06:41.604721       1 minio.go:381] Unexpected error during the creation of the csr/logging-minio-logging-csr: certificatesigningrequests.certificates.k8s.io "logging-minio-logging-csr" not found
I0228 00:06:41.604799       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-logging", Name:"logging", UID:"f6d79f13-f17f-48ca-be7e-1a3131876fa9", APIVersion:"minio.min.io/v2", ResourceVersion:"917958", FieldPath:""}): type: 'Warning' reason: 'CSRFailed' MinIO CSR Failed to create: certificatesigningrequests.certificates.k8s.io "logging-minio-logging-csr" not found
I0228 00:06:46.901173       1 monitoring.go:238] 'minio-monitoring/monitoring' Can't retrieve tenant tiers: Get "https://minio.minio-monitoring.svc.cluster.local/minio/admin/v3/tier-stats": dial tcp 100.73.56.30:443: connect: connection refused
I0228 00:06:46.909616       1 monitoring.go:122] 'minio-monitoring/monitoring' Failed to get cluster health: Get "https://minio.minio-monitoring.svc.cluster.local/minio/health/cluster": dial tcp 100.73.56.30:443: connect: connection refused
I0228 00:06:52.064910       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0228 00:06:57.963352       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0228 00:07:03.109220       1 status.go:55] Hit conflict issue, getting latest version of tenant

Error on existing secret:

I0305 09:48:03.082804       1 minio.go:308] Generating private key                                                                                                                          I0305 09:48:03.082907       1 minio.go:321] Generating CSR with CN=*.logging-hl.minio-logging.svc.cluster.local                                                                             I0305 09:48:03.089835       1 csr.go:181] Start polling for certificate of csr/logging-minio-logging-csr, every 5s, timeout after 20m0s                                                     I0305 09:48:03.089904       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-logging", Name:"logging", UID:"54ccc0cb-09d3-4ac3-8de3-9db5f9e50c5f", APIVersion:"minio.min.io/v2", ResourceVersion:"9265749", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' MinIO CSR Created
I0305 09:48:08.093246       1 csr.go:207] Certificate successfully fetched, creating secret with Private key and Certificate
E0305 09:48:08.103057       1 minio.go:392] Unexpected error during the creation of the secret/logging-tls: secrets "logging-tls" already exists

Minio internal secret generation seems flaky, we should instead rely on external certificate generation like we already do for its ingress, as docuemented here: https://min.io/docs/minio/kubernetes/aks/operations/cert-manager/cert-manager-tenants.html#create-a-certificate-for-the-tenant