CODEOWNERS blob fetch during CI variable expansion fails job delivery on Gitaly deadline (scheduler_failure, 0-byte trace)

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

A Gitaly GRPC::DeadlineExceeded while fetching the CODEOWNERS blob during CI variable expansion causes POST /api/v4/jobs/request to drop the build as scheduler_failure after ~12 seconds, with a 0-byte trace. The job is never delivered to a runner.

On 2026-08-13 this produced 1,409 occurrences on gitlab-org/gitlab alone, and was the dominant cause of that day's CI-health spike, where no_trace_available reached 11.9% of all analysed job failures (surfaced in gitlab-org/gitlab#616307).

Call path

Ci::RegisterJobService#present_build!
  → Ci::BuildRunnerPresenter#runner_variables
  → EE::MergeRequest#merge_request_approval_variables
  → ApprovalState#approved? → all_approval_rules_approved?
  → ApprovalWrappedCodeOwnerRule#branch_requires_code_owner_approval?
  → Gitlab::CodeOwners.optional_section? → CodeOwners::Loader#code_owners_file
  → EE::Repository#code_owners_blob → Repository#blobs_at
  → Gitlab::Git::Repository#batch_blobs → Gitaly
  → GRPC::DeadlineExceeded

Unlike the Rack::Timeout case in #591467, GRPC::DeadlineExceeded is a StandardError, so process_build's existing rescue StandardError catches it and drops the build via scheduler_failure!. The handling works as designed — the problem is that a transient Gitaly deadline on an approval-rule lookup is allowed to fail job delivery at all.

Evidence

Verified against 20 sampled jobs from the event: 20/20 failed with failure_reason=scheduler_failure, durations 10.4–13.6s, all returning 0 bytes from the trace endpoint. That matches the median of 11.6s reported for the scheduler_failure cohort in #616307.

Pipeline-level confirmation via the jobs API — these pipelines are almost purely this failure:

Pipeline scheduler_failure jobs
2757688679 100 / 100 failed
2756940824 50 / 52 failed (other 2 script_failure)

Daily counts, code_owners-frame Gitaly deadlines on jobs/request for gitlab-org/gitlab, 2026-08-05→08-17 — only two days have any:

Date Count
2026-08-12 96
2026-08-13 1,409

Concentrated 09:00–18:00 UTC and spread across at least 8 pipelines (top three: 195, 141, 118 jobs), so it is not attributable to a single merge request.

Controlled for instrumentation artifact: register_job_service-frame exception records are present in volume on every day of the window (1,354–8,444/day), and Rack::Timeout records appear every day. Only the code_owners + Gitaly-deadline combination is absent outside 08-12/08-13, so the zeros are real rather than a retention or indexing gap. Fleet-wide the shape is the same (08-12 = 99, 08-13 = 1,448, with 1–2 stragglers from unrelated projects), which makes this a gitlab-org/gitlab-centred event rather than a global Gitaly problem.

Why this matters beyond the one day

  1. It destroys the evidence needed to diagnose anything else. The job produces no trace, so CI-health tooling cannot analyse it. Per #616307, five bot-opened CI-health incidents in one week closed with blank root-cause tables for exactly this reason, and ci-alerts has had to raise this category's alert threshold to ≥50 pipelines to keep the noise down — which also buries the signal.
  2. The failure is misattributed. It presents as a runner/trace problem and was escalated to Runners Platform, but no runner is involved: the job never leaves the coordinator.
  3. A transient Gitaly deadline should not be fatal to job delivery. Approval-rule state is not required to hand a build to a runner.

Proposed fix

Make the CODEOWNERS lookup on this path resilient rather than fatal:

  1. Bound the CODEOWNERS blob fetch during variable expansion with a short budget and fail open — if approval state cannot be resolved in time, deliver the job rather than dropping it.
  2. If the build genuinely cannot be presented, prefer leaving it pending and returning 409 via the existing ResultFactory.invalid path so another runner retries, instead of dropping it to scheduler_failure on a transient infrastructure error.
  3. Consider whether merge_request_approval_variables needs to resolve approved? eagerly during job presentation at all, given it triggers repository I/O on the hot path.

Note that per-request memoization already exists (RequestStore in CodeOwners::Loader#code_owners_file), so this is not a missing-cache problem — the cold first fetch inside the request is what fails. Adding caching would not address it.

Relationship to #591467 and #606211 (closed)

This is the same architectural exposure as #591467 — present_build! performing unbounded external I/O inside the job-request window — reached through Gitaly rather than object storage, and exiting via a caught exception rather than a Rack::Timeout.

Worth fixing together with #591467, since a general guard around presentation-phase I/O would likely cover both. Filed separately because the exit path, failure reason, timing signature and remediation differ, and because this one has a specific measurable production event attached.

Also related: #606211 (closed) (closed) documented the same 60s/no-trace signature via a different caller; #324370 (closed) and #569685 (closed) (both closed) and #326548 (open) are the same recurring pattern of present_build! doing too much work on the hot path.

Open question

What made 2026-08-13 Gitaly-deadline-prone on this repository? The spread across many pipelines points at a Gitaly-side condition or a CODEOWNERS-blob-size threshold rather than any single change. Identifying it would confirm whether other large repositories with sizeable CODEOWNERS files are exposed.

Investigation: Runners Platform, via pubsub-rails-inf-gprd* and the GitLab API. Full notes and queries in notes/2026-08-17-no-trace-coordinator-mechanisms.md.

Edited by 🤖 GitLab Bot 🤖