## Description When a job's `secrets:` can't be resolved because of how the job or the secrets provider is configured — say the provider returns `403 permission denied` for the requested path, `400 missing role`, or a `404` for a path that doesn't exist — the runner records the job as `runner_system_failure`. That's the wrong bucket. Nothing on the runner or its host actually broke. The runner authenticated to the provider and made the request fine; the provider refused on purpose, because the secret path (or the policy behind it) that the job declared doesn't grant access. The only people who can fix that are the job owner and whoever owns the project's secrets policy. A runner operator can't do anything about it. ### Why it matters `runner_system_failure` is the reason platform operators alert and page on. It's meant to say "the runner platform is broken." When a user's secret-permission problem lands in that bucket, a few things go wrong at once. Operators get paged for something they can't fix. One project pointing at the wrong Vault path is enough to push a shared runner shard past its error SLO and open an infrastructure incident, even though every part of the fix lives in that project's CI config. The metric itself gets less trustworthy the more user-config denials we mix into it. And the job owner — the one person who can actually fix this — doesn't get a clear signal that it's theirs to fix, because we've framed it as a system failure rather than a configuration problem. ### How the misclassification happens `Build.Run` calls `resolveSecrets` and, on error, returns that error as-is. The error the secret resolvers hand back is a plain wrapped `error`, not a `*BuildError` carrying a `FailureReason`. Over in `setTraceStatus`, anything that can't be unwrapped to a `*BuildError` falls through to the default and gets reported as `RunnerSystemFailure` (the doc comment on `setTraceStatus` in `common/build.go` spells this out). So this was never a deliberate call to treat secret failures as system failures — the errors are just untagged, and untagged means "system failure" by default. ### Real-world example We opened an internal GitLab.com incident when jobs in one project started failing en masse with: ``` ERROR: Job failed (system failure): resolving secrets: reading secret: reading from Vault: api error: status code 403: permission denied ``` The jobs were picked up and submitted to the coordinator normally, then died during Vault secret resolution: the job's `secrets:` referenced a Vault path its JWT role had no policy for (an undefined CI variable had left a bogus path). Because these came through as `runner_system_failure`, they breached the shard's `ci_runner_jobs` error SLO and pulled in an infrastructure incident — for something that was entirely a project CI-config mistake. ## Proposal Classify secret-resolution failures by what actually caused them, and keep the current fail-safe default for anything we can't confidently attribute: - Provider **client errors** that point at a configuration or authorization problem (HTTP `400`, `401`, `403`, `404`) → `runner_configuration_error`, which maps to `script_failure` on GitLab instances that don't support it natively. This is the documented home for "CI configuration errors … that can only be determined by runner (and not by Rails)". - Provider **server-side failures** (HTTP `5xx`) → `runner_external_dependency_failure` (maps to `runner_system_failure`), since a genuinely unavailable secrets provider really is an external dependency operators may want to know about. - **Everything else** — network errors, timeouts, `429`, anything we can't read a status code from — stays exactly as it is today: `runner_system_failure`. If we can't attribute it confidently, the platform should still get the signal. We'd rather fail toward paging ourselves than quietly blame the user. For the mechanism, resolvers wrap their failures in small semantic error types (`ResolvingConfigurationError`, `ResolvingExternalDependencyError`) that `common/build.go` maps to the right `FailureReason`. It's generic, but this first pass only wires up the Vault resolver, since that's what we saw in the incident. AWS, GCP, Azure, and GitLab Secrets Manager can adopt the same pattern later. ### Scope and known follow-ups (deliberately left out here) - **Other secret backends** (AWS/GCP/Azure/GitLab SM): same mechanism, separate MRs. - **Retry behaviour**: `attemptResolveSecrets` retries every failure, and retrying a `403` is pointless. Left alone here to keep the change small. - **SLO robustness**: this fixes one class of misattributed failure. It doesn't, on its own, stop every user-caused failure from reaching an operator SLO — that needs per-project error attribution, which is its own piece of work. ### Naming note for reviewers `runner_configuration_error` is the least-wrong reason that already exists. Its documented scope covers CI configuration errors, and it maps to `script_failure` (the user bucket) for backward compatibility, which is exactly the routing we want. The catch is the name: it reads like a runner-manager config problem, which is a bit misleading for a job's secret-access denial. If maintainers would rather have something more precise — say `job_configuration_error` or `secret_resolution_failure` — that seems reasonable, but it needs Rails-side support and doesn't change the operator-vs-user routing, so I don't think it should block this fix. ## Impact on GitLab SaaS availability INC-11690 (GitLab.com, `private` runner shard): jobs in a single project failed en masse on Vault `403 permission denied`. Because they surfaced as `runner_system_failure`, they breached the shard's `ci_runner_jobs` error SLO and opened an infrastructure incident that took operator time to investigate — even though the fix lived entirely in the project's CI config, and no operator action could have resolved it. Every future user-config secret-permission failure carries the same risk of pulling operators into an incident they can't act on. ## How to check whether this is still a problem Trigger a job whose `secrets:` references a secrets-provider path its identity has no policy for (e.g. a Vault path with no matching JWT-role policy). Before the fix, the job fails with `failure_reason=runner_system_failure`; after the fix, it fails with `failure_reason=runner_configuration_error`. On the metrics side, confirm such failures no longer count toward the `ci_runner_jobs` `runner_system_failure` SLI. ## Links to related issues and merge requests / references - Production incident: INC-11690 (GitLab.com `private` shard `ci_runner_jobs` SLO breach) - Implementation MR: (link once created) - Prior art: gitlab-org/gitlab-runner#29295 (separate failure reason for image pull failures) - `setTraceStatus` default-to-system-failure behaviour: `common/build.go` - Failure reason definitions & compatibility map: `common/network.go`