## Context With https://gitlab.com/groups/gitlab-org/-/work_items/19739+ and https://gitlab.com/groups/gitlab-org/-/work_items/19660+ we've implemented the foundation for the Job Router as described in https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/runner_job_router (Phase 1). The Job Router is implemented as part of GitLab Relay (previously KAS) and already deployed to the GitLab.com environments (including gprd and gstg) but currently remains inactive. This is because: - `job_router` feature flag is disabled, see https://gitlab.com/gitlab-org/gitlab/-/issues/583449+ - Runner Admission Controller feature flags are disabled, see - `job_router_admission_control` - https://gitlab.com/gitlab-org/gitlab/-/issues/584394+ - `job_router_instance_runners` - https://gitlab.com/gitlab-org/gitlab/-/work_items/583577+ - The hosted runners are not deployed with the `FF_USE_JOB_ROUTER` feature flag. ## Goal The goal is to strategically rollout the Job Router to the GitLab.com environments including the hosted runners. This will include things like (including open questions we need to address): - How to rollout a runner feature flag? - The Job Router feature flag on the runner MUST be set via environment variable, see https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/39280+. - What do we need to change in runbooks? - How to test this in pre-production environments? - What do we need to change in our hosted runner setup? Including observability etc - For example, if runners are no longer hitting the public REST API endpoints to acquire jobs but the Job Router does this decrease trigger alerts? - Do we need additional metrics, dashboards, alerts? - What about SLI for job router / GitLab Relay (previously KAS) - do we need a change there? Or in the ones from hosted runners (if that exists in that form) ### Success Metrics - All hosted runners use the Job Router by default (deployed with `FF_USE_JOB_ROUTER=true`). - The Job Router is stable and jobs don't fail because of it. - The above is observed for at least a milestone (timeframe is up for debate and is likely less).