<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=616383) </details> <!--IssueSummary end--> ## Summary A Gitaly `GRPC::DeadlineExceeded` while fetching the CODEOWNERS blob during CI variable expansion causes `POST /api/v4/jobs/request` to drop the build as `scheduler_failure` after ~12 seconds, with a 0-byte trace. The job is never delivered to a runner. On 2026-08-13 this produced **1,409** occurrences on `gitlab-org/gitlab` alone, and was the dominant cause of that day's CI-health spike, where `no_trace_available` reached 11.9% of all analysed job failures (surfaced in gitlab-org/gitlab#616307). ## Call path ``` Ci::RegisterJobService#present_build! → Ci::BuildRunnerPresenter#runner_variables → EE::MergeRequest#merge_request_approval_variables → ApprovalState#approved? → all_approval_rules_approved? → ApprovalWrappedCodeOwnerRule#branch_requires_code_owner_approval? → Gitlab::CodeOwners.optional_section? → CodeOwners::Loader#code_owners_file → EE::Repository#code_owners_blob → Repository#blobs_at → Gitlab::Git::Repository#batch_blobs → Gitaly → GRPC::DeadlineExceeded ``` Unlike the `Rack::Timeout` case in #591467, `GRPC::DeadlineExceeded` **is** a `StandardError`, so `process_build`'s existing `rescue StandardError` catches it and drops the build via `scheduler_failure!`. The handling works as designed — the problem is that a transient Gitaly deadline on an approval-rule lookup is allowed to fail job delivery at all. ## Evidence Verified against 20 sampled jobs from the event: **20/20** `failed` with `failure_reason=scheduler_failure`, durations **10.4–13.6s**, all returning 0 bytes from the trace endpoint. That matches the median of 11.6s reported for the `scheduler_failure` cohort in #616307. Pipeline-level confirmation via the jobs API — these pipelines are almost purely this failure: | Pipeline | `scheduler_failure` jobs | |---|---| | 2757688679 | 100 / 100 failed | | 2756940824 | 50 / 52 failed (other 2 `script_failure`) | Daily counts, `code_owners`-frame Gitaly deadlines on `jobs/request` for `gitlab-org/gitlab`, 2026-08-05→08-17 — only two days have any: | Date | Count | |---|---:| | 2026-08-12 | 96 | | **2026-08-13** | **1,409** | Concentrated 09:00–18:00 UTC and spread across at least 8 pipelines (top three: 195, 141, 118 jobs), so it is not attributable to a single merge request. Controlled for instrumentation artifact: `register_job_service`-frame exception records are present in volume on **every** day of the window (1,354–8,444/day), and `Rack::Timeout` records appear every day. Only the `code_owners` + Gitaly-deadline combination is absent outside 08-12/08-13, so the zeros are real rather than a retention or indexing gap. Fleet-wide the shape is the same (08-12 = 99, 08-13 = 1,448, with 1–2 stragglers from unrelated projects), which makes this a `gitlab-org/gitlab`-centred event rather than a global Gitaly problem. ## Why this matters beyond the one day 1. **It destroys the evidence needed to diagnose anything else.** The job produces no trace, so CI-health tooling cannot analyse it. Per #616307, five bot-opened CI-health incidents in one week closed with blank root-cause tables for exactly this reason, and `ci-alerts` has had to raise this category's alert threshold to ≥50 pipelines to keep the noise down — which also buries the signal. 2. **The failure is misattributed.** It presents as a runner/trace problem and was escalated to Runners Platform, but no runner is involved: the job never leaves the coordinator. 3. **A transient Gitaly deadline should not be fatal to job delivery.** Approval-rule state is not required to hand a build to a runner. ## Proposed fix Make the CODEOWNERS lookup on this path resilient rather than fatal: 1. Bound the CODEOWNERS blob fetch during variable expansion with a short budget and **fail open** — if approval state cannot be resolved in time, deliver the job rather than dropping it. 2. If the build genuinely cannot be presented, prefer leaving it `pending` and returning 409 via the existing `ResultFactory.invalid` path so another runner retries, instead of dropping it to `scheduler_failure` on a transient infrastructure error. 3. Consider whether `merge_request_approval_variables` needs to resolve `approved?` eagerly during job presentation at all, given it triggers repository I/O on the hot path. Note that per-request memoization **already exists** (`RequestStore` in `CodeOwners::Loader#code_owners_file`), so this is not a missing-cache problem — the cold first fetch inside the request is what fails. Adding caching would not address it. ## Relationship to #591467 and #606211 This is the same architectural exposure as gitlab-org/gitlab#591467 — `present_build!` performing unbounded external I/O inside the job-request window — reached through Gitaly rather than object storage, and exiting via a caught exception rather than a `Rack::Timeout`. Worth fixing together with #591467, since a general guard around presentation-phase I/O would likely cover both. Filed separately because the exit path, failure reason, timing signature and remediation differ, and because this one has a specific measurable production event attached. Also related: #606211 (closed) documented the same 60s/no-trace signature via a different caller; #324370 and #569685 (both closed) and #326548 (open) are the same recurring pattern of `present_build!` doing too much work on the hot path. ## Open question What made 2026-08-13 Gitaly-deadline-prone on this repository? The spread across many pipelines points at a Gitaly-side condition or a CODEOWNERS-blob-size threshold rather than any single change. Identifying it would confirm whether other large repositories with sizeable CODEOWNERS files are exposed. --- _{Investigation: Runners Platform, via `pubsub-rails-inf-gprd*` and the GitLab API. Full notes and queries in `notes/2026-08-17-no-trace-coordinator-mechanisms.md`.}