<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=607450) </details> <!--IssueSummary end--> # Parallel Test Balancing (PTB) — API Design for the REST API layer of Parallel Test Balancing. Builds on the storage schema from gitlab-org/gitlab#607420. This issue covers the **API layer only** and assumes the schema (tables, partitioning, models) lands separately. ## Overview Three project-scoped REST endpoints authenticated by `CI_JOB_TOKEN`, called by the running parallel job itself. Because the token resolves to the `Ci::Build`, all identifying context is derived server-side and never trusted from the client: | Schema needs | Derived from the job (server-side) | |---|---| | `pipeline_id` | `job.pipeline.id` | | `pipeline_created_at` (partition key) | `job.pipeline.created_at` | | `job_group` name | `job.name` (arbitrary CI job name, interned) | | `node_index` (`CI_NODE_INDEX`) | `job.options[:instance]` | | `node_total` (`CI_NODE_TOTAL`) | parallel value from `job.options` | | retry detection | rows already stamped for `(pipeline_id, pipeline_created_at, group_id, node_index)` | ## Confirmed design decisions - **Seeding (model A):** every parallel node seeds its own static file list as `node_index = NULL` into one combined pending pool, deduped via `INSERT ... ON CONFLICT DO NOTHING`. The static per-node split is the client-side fallback if `/initialize` fails. - **`/initialize` returns a first batch** (seed + immediate claim). Claiming from a partially-seeded pool is acceptable — slowest-first budgeting means later claims naturally pick up the rest as the pool fills. - **Retry = rows already stamped for the node index** — data-driven, independent of `Ci::Build#retried?`, so it also covers crashed jobs (the assignment is the durable record). - **Empty `files` array = queue drained = stop looping.** - **Client sends durations** at seed time (snapshotted per the schema). - **Single `admin_test_balancing` job-token policy** — both endpoints mutate the queue, so there is no pure-read consumer. - **Project-actor feature flag, experiment lifecycle, `404` when off.** - **`422` for non-parallel jobs** (no `CI_NODE_INDEX`). - **Budget knobs:** only `target_duration` is an API parameter (optional, default `600`, validated `> 0`, capped). `floor_duration` (`120`) and the divisor `c` (`3`) are server constants — `floor` optimizes for API round-trip latency (an instance property, not a workload property), and `c` is internal taper tuning. ## Budget formula ``` budget = clamp(sum(pending.duration) / (node_total * c), floor, target) ``` Evaluated at each claim. Suggested defaults: `target ~= 600s`, `floor ~= 120s`, `c ~= 3`. ## Endpoint contracts ### `POST /projects/:id/ci/test_balancing/initialize` - **Auth:** `route_setting :authentication, job_token_allowed: true`; `route_setting :authorization, job_token_policies: :admin_test_balancing`. - **Params:** - `files` — `Array[{ path: String, duration: Float (nullable) }]`, bounded length. - `target_duration` — Float, optional, default `600`, validated `> 0`, capped (e.g. `<= 3600`). - **Flow:** 1. `validate_current_authenticated_job`; return `422` if the job is not parallel (no `CI_NODE_INDEX`). 2. **Retry check** — if rows exist for `(pipeline_id, pipeline_created_at, group_id, node_index)`, return `{ mode: "retry", files: [replay set] }` via the retry index (single-partition, index-only). Ignore `files`. 3. **Seed** — get-or-create group + paths; bulk `INSERT ... ON CONFLICT (pipeline_id, group_id, file_id) DO NOTHING` with `node_index = NULL`, stamped with `pipeline_created_at`. 4. **First claim** — run the same `SKIP LOCKED` budgeted claim as `/request`. 5. Return `{ mode: "seed", files: [first batch] }`. Status `201`. ### `POST /projects/:id/ci/test_balancing/request` - **Auth:** same policy. - **Params:** `target_duration` (same as above). - **Flow:** atomic `FOR UPDATE ... SKIP LOCKED` claim scoped to `(pipeline_id, pipeline_created_at, group_id)`, slowest-first, until the duration budget is met; stamp `node_index`. Return `{ files: [...] }` — an empty array signals the queue is drained. Status `201`. ### Entities `Ci::TestBalancing::Batch` -> `{ mode, files: [Ci::TestBalancing::File { path, duration }] }`. Every field typed via the `documentation:` hash. ## Files to create/modify | File | Purpose | |---|---| | `lib/api/ci/test_balancing.rb` | Grape API class (mount in `lib/api/api.rb`) | | `lib/api/entities/ci/test_balancing/batch.rb`, `file.rb` | Response entities | | `app/services/ci/test_balancing/seed_service.rb` | Get-or-create group+paths, bulk insert pending rows | | `app/services/ci/test_balancing/claim_service.rb` | `SKIP LOCKED` budgeted claim | | `app/services/ci/test_balancing/replay_service.rb` | Retry read of a node's claimed set | | `config/authz/permissions/test_balancing/admin.yml` | Raw permission (via `bin/permission`) | | `config/authz/permission_groups/assignable_permissions/ci_cd/test_balancing/admin.yml` | Assignable permission | | `lib/ci/job_token/policies.rb` + `app/validators/json_schemas/ci_job_token_policies.json` + `app/assets/javascripts/token_access/constants.js` | New `admin_test_balancing` policy (all 3 locations, per GPAT) | | `config/feature_flags/experiment/ci_parallel_test_balancing.yml` | Project actor, default off | | `spec/requests/api/ci/test_balancing_spec.rb` + service specs | Tests | | `doc/api/...` + changelog | Client-facing docs | ## Cross-cutting requirements (GitLab principles) - **Security:** all identifiers token-derived; new job-token policy is opt-in and off by default; tag `@gitlab-com/gl-security/product-security/appsec` for review before merging the policy. - **Database:** `FOR UPDATE SKIP LOCKED` lives in the service layer; partition pruning via `pipeline_created_at` equality on every query; add an `ActiveRecord::QueryRecorder` N+1 test for the batch endpoints. - **Grape:** full `desc` / `detail` / `success` / `failure` / `tags` on each endpoint; bound the `files` array and validate all params; `route_setting :lifecycle, :experiment`. - **Testing:** `'enforcing job token policies'` shared example (`expected_success_status: :created`), retry path, drained queue, non-parallel `422`, seed-dedup concurrency. - **Docs + changelog** for the client-facing API. ## Related - Schema: gitlab-org/gitlab#607420