Add Kyverno Policies for automatic MinIO pod rollout on certificate renewal

What does this MR do and why?

This MR was raised to address the issue stated in #2225 (closed). It adds two new Kyverno Policies that ensure that MinIO pods are automatically restarted when their TLS certificates are renewed. Since we now manage MinIO certificates with cert-manager, we need a mechanism to ensure that running pods pick up newly renewed certificates without manual intervention.

The policies work by monitoring certificate secrets in three key areas: the MinIO operator and both tenant namespaces (monitoring and logging). When a certificate change is detected, the policies update the appropriate annotations on either the operator Deployment or tenant StatefulSets, which triggers Kubernetes to perform a rolling restart of the affected pods.

For the MinIO operator, the policy updates the pod template annotations when the operator certificates change, causing the operator deployment to roll out new pods with the updated certificates. For the MinIO tenants, the policies are updating the spec.template.metadata annotations in the StatefulSet of the corresponding tenant, which causes the operator to reconcile the resources and roll out the tenant pods with updated certificates.

Related reference(s)

Closes #2225 (closed).

Test coverage

This was tested in a CAPO deployment which has all three MinIO elements deployed (the operator and both monitoring and logging tenants). The testing strategy was to simply delete the associated secret that is holding the certificates. After deleting the secret, the pods are getting recreated with the new secret.

Click to open test results

For minio-operator:

k get po -n minio-operator
NAME                              READY   STATUS    RESTARTS   AGE
minio-operator-6d96748677-lqqs5   1/1     Running   0          10m
minio-operator-6d96748677-pk9g4   1/1     Running   0          10m
root@BootstrapVM:/home/ubuntu/work/restart-minio/sylva-core(dg/restart-minio-on-cert-renewal)# k delete secret -n minio-operator operator-ca-tls
secret "operator-ca-tls" deleted
root@BootstrapVM:/home/ubuntu/work/restart-minio/sylva-core(dg/restart-minio-on-cert-renewal)# k get po -n minio-operator
NAME                              READY   STATUS    RESTARTS   AGE
minio-operator-566bcf7d58-dp9s6   1/1     Running   0          6s
minio-operator-566bcf7d58-szxz5   1/1     Running   0          8s

For minio-monitoring (similar to minio-logging):

k get po -n minio-monitoring
NAME                  READY   STATUS    RESTARTS   AGE
monitoring-pool-0-0   2/2     Running   0          9m10s
monitoring-pool-0-1   2/2     Running   0          9m26s
monitoring-pool-0-2   2/2     Running   0          9m45s
root@BootstrapVM:/home/ubuntu/work/restart-minio/sylva-core(dg/restart-minio-on-cert-renewal)# k delete secret -n minio-monitoring minio-monitoring-internal-tls
secret "minio-monitoring-internal-tls" deleted
root@BootstrapVM:/home/ubuntu/work/restart-minio/sylva-core(dg/restart-minio-on-cert-renewal)# k get po -n minio-monitoring
NAME                  READY   STATUS    RESTARTS   AGE
monitoring-pool-0-0   2/2     Running   0          15s
monitoring-pool-0-1   2/2     Running   0          32s
monitoring-pool-0-2   2/2     Running   0          43s

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	capd, capo, capm3
🚀	Bootstrap Provider	kubeadm (alias kadm), rke2
🐧	Node OS	ubuntu, suse
🛠️	Deployment Options	light-deploy, dev-sources, ha, misc, maxsurge-0
🎬	Pipeline Scenarios	Available scenario list and description

• 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu

• 🎬 preview ☁️ capo 🚀 rke2 🐧 suse

• 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu

• ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu

• ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse

• ☁️ capo 🚀 rke2 🐧 suse

• ☁️ capo 🚀 kadm 🐧 ubuntu

• ☁️ capo 🚀 rke2 🎬 rolling-update 🛠️ ha 🐧 ubuntu

• ☁️ capo 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu

• ☁️ capo 🚀 rke2 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 suse

• ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.3.x 🛠️ ha,misc 🐧 ubuntu

• ☁️ capm3 🚀 rke2 🐧 suse

• ☁️ capm3 🚀 kadm 🐧 ubuntu

• ☁️ capm3 🚀 kadm 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 ubuntu

• ☁️ capm3 🚀 rke2 🎬 wkld-k8s-upgrade 🛠️ ha 🐧 suse

• ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 ubuntu

• ☁️ capm3 🚀 rke2 🎬 sylva-upgrade-from-1.3.x 🛠️ misc,ha 🐧 suse

• ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 suse

Global config for deployment pipelines

• autorun pipelines
• allow failure on pipelines
• record sylvactl events

Notes:

• Enabling autorun will make deployment pipelines to be run automatically without human interaction
• Disabling allow failure will make deployment pipelines mandatory for pipeline success.
• if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.