Set clientRateLimit and clientRateLimitQPS on Kyverno admissionController (!8248) · Merge requests · Sylva-projects / sylva-core

What does this MR do and why?

This is only a hypothesis for now, as I'm not able to reproduce the issue locally at the moment (So MR still in DRAFT)

Close: #4212 (closed)

Logs on Kyverno-admission-controller :

[90m2026-06-11T20:03:34Z[0m [34mTRC[0m [1mk8s.io/client-go@v0.35.4/rest/request.go:752[0m[36m >[0m Waited before sending request [36mURL=[0mhttps://100.73.0.1:443/apis/cluster.x-k8s.io/v1beta1 [36mdelay=[0m1.007139114s [36mlogger=[0mklog [36mreason=[0m"client-side throttling, not priority and fairness" [36mv=[0m2 [36mverb=[0mGET
[90m2026-06-11T20:03:34Z[0m [32mINF[0m [1mgithub.com/kyverno/kyverno/pkg/clients/dclient/discovery.go:137[0m[36m >[0m [1mDiscovery cache invalidated after CRD update[0m [36mlogger=[0mdynamic-client/crd-definition-watcher [36mv=[0m0

We suspect the following behavior:

Because of the CRD watcher in the Kyverno Helm chart every time a new CRD is added or modified Kyverno’s discovery cache is invalidated and refreshed. This likely causes temporary slowness in Kyverno.

At the same time we observe issues with keycloak-resources-manager which fails when trying to reach the Kyverno webhook endpoint (failed to contact Kyverno webhook endpoint).

We already know that crossplane-provider-keycloak creates a large number of CRDs. This component is deployed just before keycloak-resources-manager. As a result the deployment of crossplane-provider-keycloak appears to slow Kyverno because of repeated cache invalidations triggered by the CRD watcher. During this period Kyverno may become temporarily unavailable or respond too slowly, causing keycloak-resources-manager to timeout.

According to the Kyverno documentation one possible mitigation for client-side throttling is to increase clientRateLimitBurst and clientRateLimitQPS parameters (https://kyverno.io/docs/guides/troubleshooting/#client-side-throttling). Increasing these values may help reduce throttling and improve Kyverno responsiveness during periods of heavy CRD activity.

The current values of clientRateLimitBurst is 200 and ClientRateLimitQPS is 100 :

[90m2026-06-20T19:26:59Z[0m [34mTRC[0m [1mgithub.com/kyverno/kyverno/cmd/internal/flag.go:333[0m[36m >[0m [36mclientRateLimitBurst=[0m200 [36mlogger=[0msetup/flag [36mv=[0m2
[90m2026-06-20T19:26:59Z[0m [34mTRC[0m [1mgithub.com/kyverno/kyverno/cmd/internal/flag.go:333[0m[36m >[0m [36mclientRateLimitQPS=[0m100 [36mlogger=[0msetup/flag [36mv=[0m

My proposition is to increases those values to 300 as mentionned on the Kyverno documentation : https://kyverno.io/docs/installation/customization/#container-flags

Since this change, the following logs are no longer present on admission controller logs :

[90m2026-06-11T20:03:34Z[0m [34mTRC[0m [1mk8s.io/client-go@v0.35.4/rest/request.go:752[0m[36m >[0m Waited before sending request [36mURL=[0mhttps://100.73.0.1:443/apis/cluster.x-k8s.io/v1beta1 [36mdelay=[0m1.007139114s [36mlogger=[0mklog [36mreason=[0m"client-side throttling, not priority and fairness" [36mv=[0m2 [36mverb=[0mGET

Test coverage

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	`capd`, `capo`, `capm3`
🚀	Bootstrap Provider	`kubeadm` (alias `kadm`), `rke2`, `okd`, `ck8s`
🐧	Node OS	`ubuntu`, `suse`, `na`, `leapmicro`
🛠️	Deployment Options	Deployment option list and description
🎬	Pipeline Scenarios	Available scenario list and description
🟢	Enabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🔴	Disabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🏗️	Target platform	Can be used to select specific deployment environment Available platform list and description
⚡	Pipeline control	`autorun`, `manual` or `blocking`. Can be used to override global config and start a deployment pipeline the required way

Global config for deployment pipelines

autorun pipelines
allow failure on pipelines
record sylvactl events

Notes:

Enabling autorun will make deployment pipelines to be run automatically without human interaction
Disabling allow failure will make deployment pipelines mandatory for pipeline success.
if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited Jun 22, 2026 by Remi Le Trocquer

Set clientRateLimit and clientRateLimitQPS on Kyverno admissionController

What does this MR do and why?

Related reference(s)

Test coverage

CI configuration

Global config for deployment pipelines

Merge request reports