Commit 447b7f91 authored by Vishal Tak's avatar Vishal Tak Committed by Joe Burnett
Browse files

Environment Service Blueprint

parent 8a240f3b
Loading
Loading
Loading
Loading
+2 −2
Changes for content/handbook/engineering/architecture/design-documents/cells/decisions/025_separate_worker_for_jobs_request_endpoint.md: 2 added lines, 2 removed lines.
Original line number Diff line number Diff line
@@ -29,7 +29,7 @@ Implementation involves:
2. Update the deployment pipeline to deploy the two new environments for the jobs-requests endpoint.
3. Add an explicit worker route for `/api/v4/jobs/request` in Cloudflare configuration. Cloudflare's route matching behavior is from most specific to general, so this route will take precedence over the general HTTP Router route.

Once the [Job Router](../../runner_job_router/_index.md) is implemented and all runners are switched away from long-polling, we can remove this separate worker and consolidate routing back to the main HTTP Router.
Once [push-based job dispatch](https://gitlab.com/groups/gitlab-org/-/work_items/21091) replaces job polling for all runners, we can remove this separate worker and consolidate routing back to the main HTTP Router.

## Consequences

@@ -55,7 +55,7 @@ Eliminate long-polling by switching to WebSocket connections. Once a Worker arra

Not immediately viable because:

- Already under development as part of [Job Router](../../runner_job_router/_index.md).
- The existing Relay proxy still forwards polling requests to Rails.
- Will take significant time to implement in production.
- Must still support long-polling for backward compatibility.

+4 −4
Changes for content/handbook/engineering/architecture/design-documents/ci_job_telemetry/_index.md: 4 added lines, 4 removed lines.
Original line number Diff line number Diff line
@@ -175,7 +175,7 @@ resilient to future changes.
| **`otel_traces`** | The service-agnostic ClickHouse table where the OTEL Collector writes raw spans. Follows the standard OTEL schema and is shared across all instrumented services. |
| **`ci_job_telemetry_traces`** | ([Phase 3](#phase-3-insights-layer)) A ClickHouse [Materialized View](https://clickhouse.com/docs/en/sql-reference/statements/create/view#materialized-view) that projects CI-specific columns from `otel_traces` into a query-optimized schema. Fires on insert into `otel_traces`, filtering by `ServiceName`. This is the primary table queried by the [query layer](https://gitlab.com/gitlab-org/gitlab/-/issues/590589). |
| **Job Router** | The [KAS](https://docs.gitlab.com/ee/administration/clusters/kas/) module responsible for CI job scheduling and runner assignment. Emits routing spans (`router_job_pending` through `router_job_running`) in [Phase 2](#phase-2-complete-telemetry-pipeline). |
| **Capability fingerprint** | A stable hash that uniquely identifies all factors determining job-runner compatibility: tags, runner type (instance/group/project), protected status, and project access. Runners sharing the same fingerprint can handle the same set of jobs. Defined in the [Runner Job Router architecture](/handbook/engineering/architecture/design-documents/runner_job_router/). Included as a span attribute in the MVC (Rails workstream), enabling telemetry queries like "which capability groups have slow cache restores?" or per-capability-group SLOs. |
| **Capability fingerprint** | A stable hash that uniquely identifies all factors determining job-runner compatibility: tags, runner type (instance/group/project), protected status, and project access. Runners sharing the same fingerprint can handle the same set of jobs. Defined in the [Runner Job Router architecture](https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/1049575e1c7da0fda85f7cf380b4d49a8e5d69f7/content/handbook/engineering/architecture/design-documents/runner_job_router/_index.md#capability-fingerprinting). Included as a span attribute in the MVC (Rails workstream), enabling telemetry queries like "which capability groups have slow cache restores?" or per-capability-group SLOs. |

## Proposal

@@ -1926,10 +1926,10 @@ the Runner's `job_execution` span, producing the hierarchy shown in the [span hi

### Workstream: Job Router telemetry

The [Job Router](/handbook/engineering/architecture/design-documents/runner_job_router/) (KAS Job Router Module)
The [Job Router](/handbook/engineering/architecture/design-documents/environment_service/) (KAS Job Router Module)
pushes its own spans directly to the OTEL Collector using workload identity (internal trust), since it runs in trusted infrastructure.

Job Router spans would capture the [job state transitions](/handbook/engineering/architecture/design-documents/runner_job_router/#phase-3-full-job-orchestration):
Job Router spans would capture the [job state transitions](https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/1049575e1c7da0fda85f7cf380b4d49a8e5d69f7/content/handbook/engineering/architecture/design-documents/runner_job_router/_index.md#phase-3-full-job-orchestration):

- `router_job_pending`: Time waiting in queue
- `router_job_routing`: Capability fingerprint matching and runner selection
@@ -2362,7 +2362,7 @@ This capability is **not part of the current phased roadmap** (Phases 1–3). It

- Epic: [Build scalable CI job telemetry reporting](https://gitlab.com/groups/gitlab-org/quality/analytics/-/epics/22)
- Related: [Pipeline Insights epic #11835](https://gitlab.com/groups/gitlab-org/-/epics/11835)
- [GitLab Runner Job Router](/handbook/engineering/architecture/design-documents/runner_job_router/) - KAS-based intelligent job routing with capability fingerprinting
- [GitLab Runner Job Router](/handbook/engineering/architecture/design-documents/environment_service/) - KAS-based intelligent job routing with capability fingerprinting
- [Data Insights Platform](/handbook/engineering/architecture/design-documents/data_insights_platform/)
- [Self-Managed Segmentation (OAK)](/handbook/engineering/architecture/design-documents/selfmanaged_segmentation/) - Omnibus Alternative Kit for self-managed deployment
- [GitLab Observability](https://docs.gitlab.com/operations/observability/) - Automatic CI/CD pipeline instrumentation using `GITLAB_OBSERVABILITY_EXPORT`
+461

File added.

Preview size limit exceeded, changes collapsed.

+657

File added.

Preview size limit exceeded, changes collapsed.

+135
Changes for content/handbook/engineering/architecture/design-documents/environment_service/scheduler.md: 135 added lines, 0 removed lines.
Original line number Diff line number Diff line
---
title: "Environment Service: Scheduler"
description: "How the scheduler matches requests to Providers and managers, orders waiting work, and obtains admission before assignment."
toc_hide: true
---

{{< engineering/design-document-header >}}

## Overview

The **Environment Scheduler** selects a configured Provider and one of its managers for each new environment in [Environment Service](_index.md).

After assignment, environment operations and Runs go through the Provider Manager Registry, not the scheduler.

## How a request reaches a Provider

```mermaid
flowchart TB
    Create["Caller requests an environment"]
    Save["Environment API saves the request<br>and returns an environment ID"]
    Wait["Request waits for placement"]
    Match["Match Provider scopes, tags,<br>and protection rules"]
    Managers["Find connected managers supporting<br>the specification and capabilities<br>with free capacity"]
    Suitable{"Suitable capacity available?"}
    Order["Order eligible waiting requests<br>and select the next candidate"]
    Admit["Ask applicable Admission Controllers"]
    Decision{"Admission decision"}
    Fail["Fail the request<br>Report the reason to the caller"]
    Assign["Recheck current state and eligibility<br>Reserve capacity and save the assignment<br>and creation command together"]
    Problem["Report admission problem<br>Wait before retrying"]
    Saved{"Assignment saved?"}
    Deliver["Provider Manager Registry sends<br>the saved creation command"]
    Manager["Selected Provider manager receives the command"]
    Changed["Re-read current state"]
    StillWaiting{"Still waiting and unexpired?"}
    EndAttempt["Stop this scheduling attempt"]

    Create --> Save --> Wait --> Match --> Managers --> Suitable
    Suitable -->|"No"| Wait
    Suitable -->|"Yes"| Order --> Admit --> Decision
    Decision -->|"Approve"| Assign --> Saved
    Decision -->|"Reject this Provider: consider others"| Wait
    Decision -->|"Reject the request"| Fail
    Decision -->|"Missing or invalid response"| Problem
    Problem --> Wait
    Saved -->|"No: state or capacity changed"| Changed
    Changed --> StillWaiting
    StillWaiting -->|"Yes"| Wait
    StillWaiting -->|"No: assigned, canceled, failed, or expired"| EndAttempt
    Saved -->|"Yes"| Deliver --> Manager
```

Manager acceptance does not mean the environment is ready. Readiness is reported separately.

## Matching and scheduling

### Matching

The scheduler checks Provider scopes and matching rules, including tags, untagged-work acceptance, and protection for Runner environments. It then selects a manager that supports the requested environment specification and all required capabilities.

The scheduler uses the [verified authorization context](_index.md#authorized-scope-matching) for scope and protection checks. A Provider passes the scope check when one of its allowed scope IDs appears in that context's scope chain.

The Provider must not be paused. The selected manager must be connected, healthy, recovered, and have capacity.

A Provider's rules can match without a connected manager. Placement waits for a suitable manager and capacity, subject to cancellation and the waiting deadline. Authorization must still be valid when assignment is recorded.

The service configures a maximum placement wait measured from request acceptance. It stores the deadline with the request; restarts, admission retries, and assignment attempts do not reset it. If no assignment is recorded before the deadline, the request fails with a waiting-timeout reason. This limit does not bound creation after assignment.

### Ordering waiting requests

Matching determines where work can run. Scheduling decides which eligible request goes first. Each Provider has an explicit scheduling policy.

| Policy | Ordering |
|---|---|
| Fair share | Prefer opaque workload owners with fewer active environments, then older requests. |
| Oldest first | Oldest eligible request first. |

Selecting a candidate does not remove other requests from the queue. They remain waiting until assigned, canceled, or failed.

A request needing an unsupported capability, such as stop/resume, must not block later requests the manager can serve.

### Admission and assignment

Controller scopes select specific Providers or all Providers served by Environment Service. Every applicable enforced check must approve. Dry-run checks only record outcomes; disabled or unscoped controllers receive no checks. These settings do not replace caller authorization.

Each check has a service-configured timeout. Missing, invalid, or failed responses leave the request waiting with an admission-problem reason. Checks are retried within the placement deadline. An explicit request rejection fails the request; a Provider rejection permits other Providers to be considered.

Checks apply to the proposed Provider, manager, and work. Before assignment, the scheduler rechecks that the request is waiting and unexpired, the Provider and manager still meet all matching conditions, authorization and admission remain valid, and capacity is available. It saves the assignment, creation command, and reservation atomically with the cancellation check, preventing conflicting assignments.

A manager's confirmed creation rejection releases the reservation only if it created no resources. An uncanceled, unexpired request can return to waiting with its original arrival time and deadline. A lost reply retains the assignment and reservation because the environment may exist. Failed recovery reports loss and keeps cleanup pending until deletion is confirmed.

## Demand and autoscaling

The scheduler reports per-Provider demand to an **autoscaler** outside Environment Service. The autoscaler adds or removes compute, either by changing a manager's compute capacity or the number of managers under a Provider.

For each configured Provider, it reports:

- How many matching requests are waiting.
- How long the oldest request has waited.
- How much capacity is available.
- Which environment specifications and capabilities the waiting requests require.

Demand uses Provider scopes and matching rules even when no managers are connected. Environment and capability requirements tell the autoscaler which managers could serve the waiting work. These counts are potential demand, not admission approvals. Paused Providers and requests with expired authorization are excluded.

Count each request once per matching Provider, but only once in totals across Providers. Do not count it again for each manager. Waiting counts are service-wide, not per-replica.

The scheduler refreshes matches and waiting counts when requests enter or leave the waiting state, Provider rules change, or a request's verified authorization context changes or expires.

Available capacity comes from manager reports through the Provider Manager Registry. Capacity for a request includes only managers whose registration type and capabilities support it. Demand is unavailable during initial loading, rather than reporting a misleading zero.

## Coordination and recovery

PostgreSQL stores accepted requests, configuration, assignments, commands, and status updates. Each scheduler replica builds a view of waiting work and schedules onto managers connected to that replica. Assignment and capacity reservations are committed against shared state, not decided by local queues alone.

### Loading and refreshing state

At startup, each replica loads Provider and Admission Controller configuration and waiting requests. Notifications and database checks pick up changes made during loading. Placement always rechecks current records.

The Provider Manager Registry supplies current manager connections and capacity and delivers saved creation commands. Its [recovery contract](_index.md#recovering-after-reconnect) applies before new work is assigned.

### Reliable notifications

After a database change, a Redis Pub/Sub notification tells replicas which records to reread. Notifications are hints: failed publication or a lost message must not lose accepted requests or commands.

Periodic, indexed database checks find missed work, remove canceled or assigned requests from waiting queues, and refresh configuration. These checks cover active work in bounded batches, not completed history or a full reload after every notification. Notifications reduce delay; database checks provide recovery.

A repeated notification or database rediscovery updates the existing request or command; it does not count another request or authorize another creation.

## Open questions

- How long are admission approvals and rejections valid? Which authorization or configuration changes require rechecking them?
- Should active-environment counts cover one Provider or all fair-share Providers? Do creating, stopped, or failed environments count?
- How should capability-specific demand be grouped so an autoscaler can choose suitable managers when none are connected?
- How should multiple autoscalers use overlapping Provider demand without scaling the same waiting work twice?
- Should fairness be enforced across all replicas, or is approximate fairness from independent scheduling acceptable?
Loading