## Summary 19.4.0 added an unpaginated, unindexed `LIST` on `events` inside the pod-startup poll loop — [`checkPodSandboxUserNamespaceFailure`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/util.go#L263), called from [`getPodPhase`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/util.go#L225). It is controlled by `print_pod_warning_events`, which defaults to true, and fires once per poll interval (3s) for every pod that is not yet ready. On a busy cluster with a shared runner namespace, each scan takes ~5 seconds — longer than the interval at which it is reissued. With hundreds of jobs in the startup phase this exhausted the API Priority and Fairness queues for the `service-accounts` flow schema, degrading all runner API traffic on the cluster and causing ~2.5 hours of elevated CI job failures across two production clusters. Rolling back to 19.3.2 resolved it immediately. Every other Kubernetes API call added in this release is behind a feature flag. This one is not, so it shipped active to every Kubernetes executor installation. And for any installation without `FF_SUSPENDABLE_ENVIRONMENTS`, the scans **cannot** return a matching result — the work is unconditionally wasted. Both points are evidenced under [Possible fixes](#possible-fixes). ## Steps to reproduce No special job definition is required. The call is in the pod-startup path, before the job script runs, so any job on the Kubernetes executor triggers it. 1. Run GitLab Runner 19.4.0 with the Kubernetes executor against a cluster whose runner namespace holds a large events collection. Ours holds ~5x10^5 event objects — simply what tens of thousands of jobs per day produce against the default one-hour event TTL, since each pod emits ~10-20 lifecycle events (`Scheduled`, `Pulling`/`Pulled`, `Created`, `Started` per container, secret and configMap mounts, teardown). 2. Leave `print_pod_warning_events` unset (defaults true) and `FF_SUSPENDABLE_ENVIRONMENTS` unset (defaults off). 3. Use a fixed runner namespace — `namespace_per_job = false`, the zero value. 4. Submit enough concurrent jobs that a meaningful number of pods are pending at once. 5. Observe on the API server: ``` sum(rate(apiserver_request_total{resource="events",verb="LIST",scope="namespace"}[5m])) by (code) sum(rate(apiserver_flowcontrol_rejected_requests_total{priority_level="workload-low"}[5m])) by (reason) histogram_quantile(0.5, sum(rate(apiserver_request_duration_seconds_bucket{resource="events",verb="LIST",scope="namespace"}[5m])) by (le)) ``` The events LIST rate steps up on deploy and returns to baseline on rollback to 19.3.2. ## Actual behavior While a pod is pending, the runner issues one `LIST` on `events` per 3-second poll, scoped to the pod's namespace with `FieldSelector: involvedObject.name=<pod>,type=Warning`. The request carries no `Limit` and no `ResourceVersion`, and `involvedObject.name` is not an indexed field, so the API server performs a quorum range read over the namespace's entire events collection and filters server-side. The response is small; the read is not. Measured against a ~5x10^5-object collection: | `LIST events` duration | | |---|---| | p50 | **5,308 ms** | | p90 | 7,342 ms | | p99 | 9,541 ms | Because the scan outlives its own poll interval, each job in the startup loop keeps more than one in flight continuously. This cost is inherent to the collection size rather than to the incident — the same call measured 8,877 ms mean the day *before* the upgrade, on 19.3.2, where it is reached only on the failure path. 19.4.0 did not make the scan slow; it moved an already-slow scan into the hot path and began issuing it continuously. **Call rate** (`apiserver_request_total`, `resource=events, verb=LIST, scope=namespace`, core group): | | baseline (19.3.2) | 19.4.0 | after rollback | |---|---|---|---| | cluster A, code 200 | 0.30-0.36/s | 2.31-3.07/s | 0.01-0.16/s | | cluster A, code **429** | **0.00/s** | **0.51-1.92/s** | **0.00/s** | | cluster B, code 200 | 0.32-0.46/s | 2.16-4.47/s | 0.19-0.29/s | | cluster B, code **429** | **0.00/s** | **0.05-0.72/s** | **0.00/s** | A square wave on both clusters independently, stepping up within one bucket of the deploy and collapsing on rollback. Up to ~36% of the scans were themselves rejected by the saturation they contributed to. The code [discards the error](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/util.go#L269), so none of this surfaced in job logs. **API Priority and Fairness**, `service-accounts` → `workload-low`: | | cluster A | cluster B | |---|---|---| | `queue-full` rejections, peak | 72.6/s | 42.2/s | | `time-out` rejections, peak | 16.5/s | 9.6/s | | peak `current_inqueue_requests` | 805 | 616 | | `request_concurrency_limit` | 735 | 735 | | peak `current_executing_requests` | 103 | 111 | Requests were shed on **queue depth**, not seat exhaustion — seats peaked at 14% utilisation. That is the signature of a few slow requests holding queue slots, which is what a 5-second LIST is. **Collateral damage.** `GET pods` is byte-identical between the two versions and neither its call count nor its response sizes changed, yet: | | 19.4.0 | 19.3.2 baseline | |---|---|---| | `GET pods` mean duration | 1.11 s | 0.07 s | | `LIST pods` mean duration | 0.78 s | 0.02 s | Both are quorum reads sharing the same APF flow as the events scans, and both queued behind them. Since the pod GET's own code, count and payload are unchanged, the added latency is wait time. **Feedback loop.** From `gitlab_runner_jobs` by `executor_stage`: | `executor_stage=prepare`, by stage | change vs baseline | |---|---| | `state=pending` | 0 → hundreds | | `prepare_script` | **8.6x** | | `after_script` | **8.7x** | | `cleanup_file_variables` | **6.8x** | | `step_script` (actual build execution) | **0.8x** | Stages that wait on the API server grew 6.8-8.7x while actual build execution *fell*. Jobs were not doing more work; they were accumulating in the phases that poll. Slower pod startup keeps jobs in the loop longer, which means more concurrent scans, which slows startup further. **User-visible effect**, over the 2h25m window on the harder-hit of the two clusters: `runner_system_failure` rose from a 0.18% baseline to **13.69%**, and `job_execution_timeout` from 0.56% to 5.61%. Total job failure went from 8.74% to 26.58% — the baseline there is ordinary script failure, so the delta is the infrastructure-attributable part. Affected jobs stalled until killed by `activeDeadlineSeconds`, presenting to users as a job hung for the full timeout with no output. ## Expected behavior Upgrading to 19.4.0 without enabling any new feature should not change the runner's Kubernetes API call pattern. 19.3.2 issues no events LIST on the pod-startup path. Its only events LIST is in [`logPodWarningEvents`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/kubernetes.go#L750), reached only from an error branch, and it sets [`Limit: 200`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/kubernetes.go#L759). ## Relevant logs and/or screenshots The runner logs nothing for the LIST itself — the error is discarded at `util.go:269`. Everything below is downstream of the API server being saturated, and every terminal failure is timeout-shaped rather than naming a cause, which is what made this hard to attribute from the runner side: <details> <summary> job log </summary> ```sh # Terminal failures seen by users — all timeout-shaped, none naming the events LIST: ERROR: Job failed (system failure): couldn't get pod details: the server was unable to return a response in the time allotted, but may still be processing the request (get pods runner-<id>-project-<n>-concurrent-9-<hash>) failure_reason=runner_system_failure ERROR: Job failed (system failure): pod "<ns>/runner-<id>-project-<n>-concurrent-5-<hash>" is being deleted failure_reason=runner_system_failure # The pod-status poll failing, at high volume (1,322 occurrences in the sampled window): WARNING: Getting job pod status error=Get "https://<apiserver>/api/v1/namespaces/<ns>/pods/runner-<id>-project-<n>-concurrent-0-<hash>": context canceled WARNING: Getting job pod status error=client rate limiter Wait returned an error: context canceled WARNING: Getting job pod status error=the server was unable to return a response in the time allotted, but may still be processing the request (get pods runner-<id>-...) # Informer setup failing, so the executor fell back to polling — including one explicit 429: WARNING: won't use informers: "SelfSubjectAccessReview creation: the server was unable to return a response in the time allotted", see: https://docs.gitlab.com/runner/executors/kubernetes/#informers WARNING: won't use informers: "SelfSubjectAccessReview creation: the server has received too many requests", see: https://docs.gitlab.com/runner/executors/kubernetes/#informers # context deadline exceeded on teardown paths (482 occurrences): WARNING: Error while executing file based variables removal script error=context deadline exceeded WARNING: Error streaming logs <ns>/runner-<id>-.../helper:/logs-<n>/output.log: context deadline exceeded ``` </details> ## Environment description Custom installation. Kubernetes executor with the attach strategy, on managed Kubernetes. Two production clusters, high-volume CI — tens of thousands of jobs per cluster per day against a shared runner namespace, giving a standing events collection of ~5x10^5 objects. <details> <summary> config.toml contents </summary> ```toml concurrent = 200 check_interval = 10 log_level = "info" [[runners]] executor = "kubernetes" request_concurrency = 40 [runners.feature_flags] FF_CONCRETE = false # FF_SUSPENDABLE_ENVIRONMENTS not set -> off [runners.kubernetes] namespace = "<fixed-runner-namespace>" namespace_per_job = false poll_timeout = 180 # poll_interval not set -> default 3s # print_pod_warning_events not set -> default true image = "alpine:latest" pull_policy = ["if-not-present"] cpu_limit = "4500m" memory_limit = "12Gi" ``` </details> ### Used GitLab Runner version ``` Runtime platform arch=amd64 os=linux pid=1 revision=ac11717a version=19.4.0 ``` Rolled back to, and currently running: ``` Runtime platform arch=amd64 os=linux pid=1 revision=23a5dafc version=19.3.2 ``` ## Possible fixes Added by !6781 (`feat(kubernetes): add suspend/resume support via single-PVC overlayfs`), commit `bbcd5520a`. The call site, in [`getPodPhase` at `util.go:225-229`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/util.go#L225-229), placed after the early returns for "pod ready" ([`:216`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/util.go#L216)) and "terminal container error" ([`:221`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/util.go#L221)), so it runs on every iteration while the pod is pending: ```go if printPodWarningEvents { if err := checkPodSandboxUserNamespaceFailure(ctx, client, pod); err != nil { return podPhaseResponse{true, api.PodUnknown, err} } } ``` The function, [`util.go:263-283`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/util.go#L263-283): ```go func checkPodSandboxUserNamespaceFailure(ctx context.Context, client kubernetes.Interface, pod *api.Pod) error { // kubeAPI: events, list, print_pod_warning_events=true events, err := client.CoreV1().Events(pod.Namespace).List(ctx, metav1.ListOptions{ FieldSelector: fmt.Sprintf("involvedObject.name=%s,type=%s", pod.Name, k8sEventWarningType), }) if err != nil { return nil } ... } ``` The enclosing poll loop is [`waitForPodRunning`, `util.go:303`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/util.go#L303). Two observations from diffing the tags, offered as context rather than as a proposed remedy. **It is the only new API call in the release not behind a feature flag.** Using the repository's own `// kubeAPI:` annotations, the complete set of call sites added between v19.3.2 and v19.4.0: ``` // kubeAPI: persistentvolumeclaims, create, FF_SUSPENDABLE_ENVIRONMENTS=true // kubeAPI: persistentvolumeclaims, delete, FF_SUSPENDABLE_ENVIRONMENTS=true // kubeAPI: configmaps, create, FF_SUSPENDABLE_ENVIRONMENTS=true // kubeAPI: configmaps, get, FF_SUSPENDABLE_ENVIRONMENTS=true // kubeAPI: configmaps, update, FF_SUSPENDABLE_ENVIRONMENTS=true // kubeAPI: configmaps, delete, FF_SUSPENDABLE_ENVIRONMENTS=true // kubeAPI: namespaces, get, FF_SUSPENDABLE_ENVIRONMENTS=true // kubeAPI: pods, get, FF_CONCRETE=true // kubeAPI: pods/exec, ..., FF_CONCRETE=true // kubeAPI: pods/log, get, FF_CONCRETE=true // kubeAPI: events, list, print_pod_warning_events=true <-- not a feature flag ``` None were removed. The rest of !6781 is gated behind `FF_SUSPENDABLE_ENVIRONMENTS` and is inert for installations not using it. **The check cannot match unless suspend/resume is in use.** It looks only for `FailedCreatePodSandBox` events whose message contains `does not support user namespaces`, which requires `spec.hostUsers: false`. `HostUsers` is assigned in exactly one place, [`kubernetes.go:2624`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/kubernetes.go#L2624), from [`isHostUserEnabled`, `suspend.go:60`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/suspend.go#L60): ```go func (s *executor) isHostUserEnabled() *bool { if !s.usesSuspendResume() { return nil } return new(false) } ``` and [`usesSuspendResume`, `suspend.go:338`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/suspend.go#L338) returns false unless `FF_SUSPENDABLE_ENVIRONMENTS` is enabled **and** the job carries suspend options. With the flag off, `HostUsers` is `nil`, the field is omitted from the pod spec, the kubelet never rejects it, and the event being searched for cannot exist. Also relevant to any fix: the informer factory, `feature.go` (SelfSubjectAccessReview), and the pod `GET` call site are byte-identical between the two tags, `client-go` is unchanged at v0.36.3, and no retry or QPS/Burst constants changed. This call is the only behavioural difference in the executor's API usage. **Our mitigation was to roll back to 19.3.2**, which is what we are still running. We have not attempted to run 19.4.0 with `print_pod_warning_events = false`, so we cannot report on whether that is a viable alternative — though from the code the call is unreachable when it is false. Note that one setting also controls the per-poll pod `status.Conditions` output ([`util.go:242-256`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/v19.4.0/executors/kubernetes/util.go#L242-256)) and warning-event retrieval on job failure, so disabling it is not free.