Minio pods restarted by kyverno policies

Summary

This issue ha not been observed in CI, but we observed in a sylva deployment that minio pods were frequently restarted by kyverno policy that were introduced in !4117 (merged):

2025-03-27T23:59:48Z INF github.com/kyverno/kyverno/pkg/background/mutate/mutate.go:92 > processing mutate existing logger=background name=ur-qkdlm policy=minio-logging/restart-minio-logging-tenant-on-cert-renewal resource=v1/Secret/minio-logging/minio-logging-internal-tls v=0
2025-03-27T23:59:48Z TRC github.com/kyverno/kyverno/pkg/policy/mutate.go:56 > successfully created UR on policy update logger=PolicyController policy=restart-minio-logging-tenant-on-cert-renewal rule=minio-logging-tenant-restart rule type=mutate target=v1/Secret/minio-logging/minio-logging-internal-tls v=2
2025-03-28T00:00:18Z TRC github.com/kyverno/kyverno/pkg/policy/mutate.go:56 > successfully created UR on policy update logger=PolicyController policy=restart-minio-logging-tenant-on-cert-renewal rule=minio-logging-tenant-restart rule type=mutate target=v1/Secret/minio-logging/minio-logging-internal-tls v=2
2025-03-28T00:00:18Z INF github.com/kyverno/kyverno/pkg/background/mutate/mutate.go:92 > processing mutate existing logger=background name=ur-qj9fb policy=minio-logging/restart-minio-logging-tenant-on-cert-renewal resource=v1/Secret/minio-logging/minio-logging-internal-tls v=0
2025-03-28T00:00:48Z TRC github.com/kyverno/kyverno/pkg/policy/mutate.go:56 > successfully created UR on policy update logger=PolicyController policy=restart-minio-logging-tenant-on-cert-renewal rule=minio-logging-tenant-restart rule type=mutate target=v1/Secret/minio-logging/minio-logging-internal-tls v=2
2025-03-28T00:00:48Z INF github.com/kyverno/kyverno/pkg/background/mutate/mutate.go:92 > processing mutate existing logger=background name=ur-h4zqc policy=minio-logging/restart-minio-logging-tenant-on-cert-renewal resource=v1/Secret/minio-logging/minio-logging-internal-tls v=0
2025-03-28T00:01:18Z INF github.com/kyverno/kyverno/pkg/background/mutate/mutate.go:92 > processing mutate existing logger=background name=ur-q7dqw policy=minio-logging/restart-minio-logging-tenant-on-cert-renewal resource=v1/Secret/minio-logging/minio-logging-internal-tls v=0
2025-03-28T00:01:18Z TRC github.com/kyverno/kyverno/pkg/policy/mutate.go:56 > successfully created UR on policy update logger=PolicyController policy=restart-minio-logging-tenant-on-cert-renewal rule=minio-logging-tenant-restart rule type=mutate target=v1/Secret/minio-logging/minio-logging-internal-tls v=2
2025-03-28T00:01:48Z INF github.com/kyverno/kyverno/pkg/background/mutate/mutate.go:92 > processing mutate existing logger=background name=ur-wwmvm policy=minio-logging/restart-minio-logging-tenant-on-cert-renewal resource=v1/Secret/minio-logging/minio-logging-internal-tls v=0
2025-03-28T00:01:48Z TRC github.com/kyverno/kyverno/pkg/policy/mutate.go:56 > successfully created UR on policy update logger=PolicyController policy=restart-minio-logging-tenant-on-cert-renewal rule=minio-logging-tenant-restart rule type=mutate target=v1/Secret/minio-logging/minio-logging-internal-tls v=2
2025-03-28T00:02:18Z TRC github.com/kyverno/kyverno/pkg/policy/mutate.go:56 > successfully created UR on policy update logger=PolicyController policy=restart-minio-logging-tenant-on-cert-renewal rule=minio-logging-tenant-restart rule type=mutate target=v1/Secret/minio-logging/minio-logging-internal-tls v=2
2025-03-28T00:02:18Z INF github.com/kyverno/kyverno/pkg/background/mutate/mutate.go:92 > processing mutate existing logger=background name=ur-rl8hn policy=minio-logging/restart-minio-logging-tenant-on-cert-renewal resource=v1/Secret/minio-logging/minio-logging-internal-tls v=0
2025-03-28T00:02:18Z ERR github.com/kyverno/kyverno/pkg/background/mutate/mutate.go:215 > failed to update target resource error="Operation cannot be fulfilled on statefulsets.apps \"logging-pool-0\": the object has been modified; please apply your changes to the latest version and try again" logger=background/minio-logging-tenant-restart name=logging-pool-0 namespace=minio-logging policy=minio-logging/restart-minio-logging-tenant-on-cert-renewal resource=v1/Secret/minio-logging/minio-logging-internal-tls
2025-03-28T00:02:19Z INF github.com/kyverno/kyverno/pkg/background/mutate/mutate.go:92 > processing mutate existing logger=background name=ur-rl8hn policy=minio-logging/restart-minio-logging-tenant-on-cert-renewal resource=v1/Secret/minio-logging/minio-logging-internal-tls v=0

After a few hours, the generation of minio StatefulSet was fairly high:

- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    annotations:
      meta.helm.sh/release-name: minio-logging
      meta.helm.sh/release-namespace: minio-logging
      min.io/revision: "0"
      prometheus.io/path: /minio/v2/metrics/cluster
      prometheus.io/port: "9000"
      prometheus.io/scheme: http
      prometheus.io/scrape: "true"
    creationTimestamp: "2025-03-27T23:24:19Z"
    generation: 156

The only forseenable explanation is that secret's resourceVersion is being incremented for some reason. As resourceVersion will be updated as soon as there is any change in the object, it would be safer to annotate the deployments/statefulsets with a hash of secret's data that will be hopefully more stable.