tigera-operator uselessly loads kyverno

Summary

While working on kyverno, I observed high load on admission-controller:

[kyverno-admission-controller-6b7f579cdf-4wqdw kyverno] 2024-08-27T09:41:41Z    INFO    webhooks.resource.validate      validation/validation.go:125    validation passed       {"gvk": "apps/v1, Kind=Deployment", "gvr": {"group":"apps","version":"v1","resource":"deployments"}, "namespace": "calico-system", "name": "calico-typha", "operation": "UPDATE", "uid": "dc83762e-fd63-4307-b3f5-6ad36fe3c231", "user": {"username":"system:serviceaccount:tigera-operator:tigera-operator","uid":"ae3cba86-fdd7-42e6-b90d-d8bd9de8b60b","groups":["system:serviceaccounts","system:serviceaccounts:tigera-operator","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["tigera-operator-664886c699-ghzn7"],"authentication.kubernetes.io/pod-uid":["e706576b-0b50-4745-b0a1-61cc00c5f743"]}}, "roles": [], "clusterroles": ["system:basic-user", "system:discovery", "system:public-info-viewer", "system:service-account-issuer-discovery", "tigera-operator", "tigera-operator-psa"], "resource.gvk": "apps/v1, Kind=Deployment", "kind": "Deployment", "URLParams": "", "action": "validate", "resource": "calico-system/Deployment/calico-typha", "operation": "UPDATE", "gvk": "apps/v1, Kind=Deployment", "policy": "disallow-default-namespace"}
[kyverno-admission-controller-6b7f579cdf-4wqdw kyverno] 2024-08-27T09:41:41Z    INFO    webhooks.resource.validate      validation/validation.go:125    validation passed       {"gvk": "apps/v1, Kind=DaemonSet", "gvr": {"group":"apps","version":"v1","resource":"daemonsets"}, "namespace": "calico-system", "name": "calico-node", "operation": "UPDATE", "uid": "d6deb3a0-9300-4248-b740-84c6fd2a65df", "user": {"username":"system:serviceaccount:tigera-operator:tigera-operator","uid":"ae3cba86-fdd7-42e6-b90d-d8bd9de8b60b","groups":["system:serviceaccounts","system:serviceaccounts:tigera-operator","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["tigera-operator-664886c699-ghzn7"],"authentication.kubernetes.io/pod-uid":["e706576b-0b50-4745-b0a1-61cc00c5f743"]}}, "roles": [], "clusterroles": ["system:basic-user", "system:discovery", "system:public-info-viewer", "system:service-account-issuer-discovery", "tigera-operator", "tigera-operator-psa"], "resource.gvk": "apps/v1, Kind=DaemonSet", "kind": "DaemonSet", "URLParams": "", "action": "validate", "resource": "calico-system/DaemonSet/calico-node", "operation": "UPDATE", "gvk": "apps/v1, Kind=DaemonSet", "policy": "disallow-latest-and-main-tag"}
[kyverno-admission-controller-6b7f579cdf-4wqdw kyverno] 2024-08-27T09:41:41Z    INFO    webhooks.resource.validate      validation/validation.go:125    validation passed       {"gvk": "apps/v1, Kind=DaemonSet", "gvr": {"group":"apps","version":"v1","resource":"daemonsets"}, "namespace": "calico-system", "name": "calico-node", "operation": "UPDATE", "uid": "d6deb3a0-9300-4248-b740-84c6fd2a65df", "user": {"username":"system:serviceaccount:tigera-operator:tigera-operator","uid":"ae3cba86-fdd7-42e6-b90d-d8bd9de8b60b","groups":["system:serviceaccounts","system:serviceaccounts:tigera-operator","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["tigera-operator-664886c699-ghzn7"],"authentication.kubernetes.io/pod-uid":["e706576b-0b50-4745-b0a1-61cc00c5f743"]}}, "roles": [], "clusterroles": ["system:basic-user", "system:discovery", "system:public-info-viewer", "system:service-account-issuer-discovery", "tigera-operator", "tigera-operator-psa"], "resource.gvk": "apps/v1, Kind=DaemonSet", "kind": "DaemonSet", "URLParams": "", "action": "validate", "resource": "calico-system/DaemonSet/calico-node", "operation": "UPDATE", "gvk": "apps/v1, Kind=DaemonSet", "policy": "disallow-default-namespace"}
[kyverno-admission-controller-6b7f579cdf-4wqdw kyverno] 2024-08-27T09:41:41Z    INFO    webhooks.resource.validate      validation/validation.go:125    validation passed       {"gvk": "apps/v1, Kind=Deployment", "gvr": {"group":"apps","version":"v1","resource":"deployments"}, "namespace": "calico-system", "name": "calico-kube-controllers", "operation": "UPDATE", "uid": "03bf1503-fe82-41db-b0f0-b100f190aa0a", "user": {"username":"system:serviceaccount:tigera-operator:tigera-operator","uid":"ae3cba86-fdd7-42e6-b90d-d8bd9de8b60b","groups":["system:serviceaccounts","system:serviceaccounts:tigera-operator","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["tigera-operator-664886c699-ghzn7"],"authentication.kubernetes.io/pod-uid":["e706576b-0b50-4745-b0a1-61cc00c5f743"]}}, "roles": [], "clusterroles": ["system:basic-user", "system:discovery", "system:public-info-viewer", "system:service-account-issuer-discovery", "tigera-operator", "tigera-operator-psa"], "resource.gvk": "apps/v1, Kind=Deployment", "kind": "Deployment", "URLParams": "", "action": "validate", "resource": "calico-system/Deployment/calico-kube-controllers", "operation": "UPDATE", "gvk": "apps/v1, Kind=Deployment", "policy": "pdb-minavailable-check"}
[kyverno-admission-controller-6b7f579cdf-4wqdw kyverno] 2024-08-27T09:41:41Z    INFO    webhooks.resource.validate      validation/validation.go:125    validation passed       {"gvk": "apps/v1, Kind=Deployment", "gvr": {"group":"apps","version":"v1","resource":"deployments"}, "namespace": "calico-system", "name": "calico-kube-controllers", "operation": "UPDATE", "uid": "03bf1503-fe82-41db-b0f0-b100f190aa0a", "user": {"username":"system:serviceaccount:tigera-operator:tigera-operator","uid":"ae3cba86-fdd7-42e6-b90d-d8bd9de8b60b","groups":["system:serviceaccounts","system:serviceaccounts:tigera-operator","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["tigera-operator-664886c699-ghzn7"],"authentication.kubernetes.io/pod-uid":["e706576b-0b50-4745-b0a1-61cc00c5f743"]}}, "roles": [], "clusterroles": ["system:basic-user", "system:discovery", "system:public-info-viewer", "system:service-account-issuer-discovery", "tigera-operator", "tigera-operator-psa"], "resource.gvk": "apps/v1, Kind=Deployment", "kind": "Deployment", "URLParams": "", "action": "validate", "resource": "calico-system/Deployment/calico-kube-controllers", "operation": "UPDATE", "gvk": "apps/v1, Kind=Deployment", "policy": "disallow-default-namespace"}
[kyverno-admission-controller-6b7f579cdf-4wqdw kyverno] 2024-08-27T09:41:41Z    INFO    webhooks.resource.validate      validation/validation.go:125    validation passed       {"gvk": "apps/v1, Kind=Deployment", "gvr": {"group":"apps","version":"v1","resource":"deployments"}, "namespace": "calico-system", "name": "calico-kube-controllers", "operation": "UPDATE", "uid": "03bf1503-fe82-41db-b0f0-b100f190aa0a", "user": {"username":"system:serviceaccount:tigera-operator:tigera-operator","uid":"ae3cba86-fdd7-42e6-b90d-d8bd9de8b60b","groups":["system:serviceaccounts","system:serviceaccounts:tigera-operator","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["tigera-operator-664886c699-ghzn7"],"authentication.kubernetes.io/pod-uid":["e706576b-0b50-4745-b0a1-61cc00c5f743"]}}, "roles": [], "clusterroles": ["system:basic-user", "system:discovery", "system:public-info-viewer", "system:service-account-issuer-discovery", "tigera-operator", "tigera-operator-psa"], "resource.gvk": "apps/v1, Kind=Deployment", "kind": "Deployment", "URLParams": "", "action": "validate", "resource": "calico-system/Deployment/calico-kube-controllers", "operation": "UPDATE", "gvk": "apps/v1, Kind=Deployment", "policy": "disallow-latest-and-main-tag"}

It seems to be caused bu tigera-operator that is reconcilling its manageed resources in loop at faily high frequency:

{"level":"info","ts":"2024-08-27T09:39:10Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:11Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:11Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:12Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:12Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:12Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:13Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:13Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:14Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:14Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:15Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:15Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:16Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:16Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:17Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:17Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:18Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:18Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:19Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:19Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:20Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:21Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:21Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:22Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:22Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:22Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:23Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:23Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:24Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:24Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}
{"level":"info","ts":"2024-08-27T09:39:25Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-node"}
{"level":"info","ts":"2024-08-27T09:39:25Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-cni-plugin"}
{"level":"info","ts":"2024-08-27T09:39:26Z","logger":"controller_installation","msg":"Reconciling Installation.operator.tigera.io","Request.Namespace":"","Request.Name":"calico-kube-controllers"}

We observed a fairly high load (~0.5 cpu) on admission controllers that could probably be reduced:

image

After deleting tigera-operator deployment, load was much lower:

image

Edited by Francois Eleouet