## Context The `job_router` SLI in [`metrics-catalog/services/kas.jsonnet`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/services/kas.jsonnet) generates the SLO alerts, so the error and latency alerts come from the SLI rather than being hand-authored. The alert runbook is [KasServiceJobRouter](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/kas/alerts/KasServiceJobRouter.md). | Alert | Condition | Severity | Status | |-------|-----------|----------|--------| | Job Router error rate | GetJob error ratio SLO | S2 | Done: `KasServiceJobRouterErrorSLOViolation`, plus `KasServiceJobRouterTrafficAbsent` and `KasServiceJobRouterTrafficCessation`. Target 99.5% on a 98% confidence interval (gitlab-com/runbooks!11234, gitlab-com/runbooks!11392). | | Job Router latency | Apdex on `job_router_get_job_duration_seconds{result="job"}`: satisfied under 2.5 s, tolerated under 5 s, target 99% | S2 | Done: `KasServiceJobRouterApdexSLOViolation` (gitlab-com/runbooks!11863). Baseline and validation in the comment below. | | Relay→Rails failures | Rails responses to Relay by HTTP status, per call | S2 (error SLO), S4 (new SLI) | Done. No separate paging alert: every 5xx and failed connection maps to `Unavailable`, which the error SLO counts, so an outage of the internal endpoint pages through `KasServiceJobRouterErrorSLOViolation`. Attribution: GitLab's HTTP status as a log field and a counter of GitLab responses by call and status (gitlab-org/cluster-integration/gitlab-agent#1202, gitlab-org/cluster-integration/gitlab-agent!4812). The counter is the `job_router_gitlab_requests` SLI of `kas` with `KasServiceJobRouterGitlabRequestsErrorSLOViolation` (gitlab-com/runbooks!11982): S4, Slack only, out of the service aggregation so the failures are not counted twice. | | Discovery endpoint errors | `GET /api/v4/runners/router/discovery` error rate or latency | S4 | Done: gitlab-com/runbooks!11981 adds `RunnerShardFallingBackFromJobRouter` (Slack only). A failed discovery call is a `no_discovery` fallback on the runner, so the runner-side counter covers the endpoint without a Rails rule. Rails also records the endpoint in the request SLI by `endpoint_id`. | Landed alongside, in the ci-runners service (gitlab-com/runbooks!11726): `RunnerTokenRateLimitedOnJobRouter`, `RunnerTokenRateLimitedOnDirectPath`, and `RunnerManagerNotRequestingJobs` counting both paths. ## Tasks - [x] Tune the error ratio: 99.5% with a 98% confidence interval, instead of the inherited KAS default (gitlab-com/runbooks!11234, gitlab-com/runbooks!11392) - [x] gitlab-org/cluster-integration/gitlab-agent#1044 deployed: `job_router_get_job_duration_seconds` by `result`, with the dashboard panels (gitlab-com/runbooks!11283) - [x] Read a baseline for `result="job"`: p50 0.66 s, p90 0.99 s, p99 2.45 s in gprd over six days ([numbers and the proposed thresholds](https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/39391#note_4042616525)) - [x] Add the apdex to the `job_router` SLI: gitlab-com/runbooks!11863. Satisfied under 2.5 s, tolerated under 5 s, target 99%; replaces the "deliberately no apdex" comment; runbook section added. - [x] Decide the Relay→Rails signal: no separate alert, since 5xx and failed connections page through the error SLO as `Unavailable`; attribution (status as a log field, a counter of GitLab responses by call and status) is in gitlab-org/cluster-integration/gitlab-agent!4812 (gitlab-org/cluster-integration/gitlab-agent#1202); the counter is the S4 `job_router_gitlab_requests` SLI (gitlab-com/runbooks!11982) - [x] Alert on a shard falling back from the router (`no_discovery`, `dial_failed`): gitlab-com/runbooks!11981 ## Notes Routing: the SLI carries `feature_category: runner_core`, so its alerts post to `#alerts-ci-cd` and the S2 alerts page the SRE on-call through incident.io. The alerts stay with Runner Core (gitlab-com/runbooks!11966 closed); Runner Core's own paging rotation is being set up in gitlab-org/primitives#25. `grpc_server_handling_seconds` is not a usable latency basis. GetJob is a long poll, so it measures how long GitLab held the request rather than work Relay did. `job_router_get_job_duration_seconds` carries a `result` label, so the apdex can use only the calls that returned a job. ## References - SLI: https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11234, https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11392 - Dashboard: https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11283, https://dashboards.gitlab.net/d/kas-job-router - Duration metric: https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/work_items/1044, https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/4157 - Rate limit and not-requesting alerts: https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11726 - Relay→Rails visibility: https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/work_items/1202, https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/4812, https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11982 - PREP MRs: https://gitlab.com/gitlab-org/architecture/readiness/-/merge_requests/93+, https://gitlab.com/gitlab-org/architecture/readiness/-/merge_requests/170+, https://gitlab.com/gitlab-org/architecture/readiness/-/merge_requests/173+ - PREP tracking: https://gitlab.com/gitlab-org/architecture/readiness/-/work_items/100+