Draft: Add REST API for parallel test balancing

What does this MR do and why?

Adds the REST API layer for Parallel Test Balancing, implementing #607450 (closed). Builds on the storage schema from !248189 (merged) (this MR targets that branch — stacked MR, retarget to master once the schema MR merges).

Two project-scoped, CI_JOB_TOKEN-authenticated endpoints called by the running parallel: job itself. All identifying context is derived server-side from the authenticated job and never trusted from the client (project_id, pipeline_id, pipeline_created_at partition key, job group from Gitlab::Utils::Job.group_name(job.name), node_index from job.options[:instance]):

Endpoint	Behavior
POST /projects/:id/ci/test_balancing/initialize	Retry check (rows already stamped for this node → replay original set, mode: retry), else seed the shared pending pool with the caller's static split (INSERT ... ON CONFLICT DO NOTHING) and claim a first batch (mode: seed).
POST /projects/:id/ci/test_balancing/request	Atomic FOR UPDATE SKIP LOCKED budgeted claim, slowest-first. Empty files array = queue drained, stop looping.

Budget formula (evaluated per claim, in Ci::TestBalancing::ClaimService):

budget = clamp(sum(pending.duration) / (node_total * TAPER_DIVISOR), FLOOR_DURATION, target_duration)

Only target_duration is an API parameter (default 600, capped at 3600); FLOOR_DURATION (120) and TAPER_DIVISOR (3) are server constants.

Access control

• New opt-in admin_test_balancing job token policy (all 3 locations: lib/ci/job_token/policies.rb, JSON schema, frontend allowlist UI constants). Off by default, opt-in via the allowlist. ⚠️ cc @gitlab-com/gl-security/product-security/appsec — new job token permission, requesting AppSec review.
• New update_test_balancing role permission (developer+, config/authz/roles/developer.yml) as the base authorization — job token policies only narrow access, so the endpoint also requires the job user to have developer access to the target project.
• route_setting :lifecycle, :experiment + project-actor feature flag ci_parallel_test_balancing (experiment, default off, 404 when off).
• 422 for non-parallel jobs (no CI_NODE_INDEX).

Notable implementation decisions (deviations from the work item)

1. skip_granular_token_authorization: :job_token_auth instead of GPAT permissions: decorators: these endpoints are semantically job-token-only (the node identity comes from the build), following the GET /job precedent. Consequently no assignable GPAT permission YAML is created — an assignable permission never referenced by a decorator fails gitlab:permissions:validate.
2. Raw permission is update_test_balancing, not admin_test_balancing: the permission validator and Gitlab/Authz/PermissionCheck cop disallow new admin_* permissions (prefer granular actions). The job token policy keeps the admin_ name per that subsystem's read/admin convention.
3. The job-token-policies shared example runs with expected_success_status: :unprocessable_entity: the shared example's target_job is defined in a nested context (cannot be overridden) and is not a parallel job, so the "authorized" outcome is the endpoint's 422 — it still proves the 403 policy gate fires first. Direct 201 paths are covered by the endpoint specs.
4. Unknown durations are budgeted at the average known pending duration (self-tuning) and sort first (NULLS FIRST, matching the partial index order), so unpredictable files run early.
5. Claim lock bounding: a single claim locks at most min((budget/avg_duration) * 3, 2000) rows so concurrent claimers skip past the locked prefix instead of observing a spuriously empty queue.

Database

Single-statement claim (data-modifying CTE, executed on the primary; scoped by (project_id, pipeline_id, pipeline_created_at, job_group_id) — the partition key equality prunes to one partition, and every query carries the sharding key):

WITH locked AS (
  SELECT test_id, duration
  FROM ci_test_balancing_assignments
  WHERE project_id = $p AND pipeline_id = $1 AND pipeline_created_at = $2 AND job_group_id = $3
    AND node_index IS NULL
  ORDER BY duration DESC
  LIMIT $4
  FOR UPDATE SKIP LOCKED
),
budgeted AS (
  SELECT test_id,
    SUM(COALESCE(duration, $assumed)) OVER (ORDER BY duration DESC NULLS FIRST, test_id)
      - COALESCE(duration, $assumed) AS duration_before
  FROM locked
),
claimed AS (
  UPDATE ci_test_balancing_assignments assignments
  SET node_index = $5
  FROM budgeted
  WHERE assignments.project_id = $p AND assignments.pipeline_id = $1
    AND assignments.pipeline_created_at = $2 AND assignments.job_group_id = $3
    AND assignments.test_id = budgeted.test_id
    AND budgeted.duration_before < $budget
  RETURNING assignments.test_id, assignments.duration
)
SELECT tests.path, claimed.duration
FROM claimed
JOIN ci_test_balancing_tests tests ON tests.id = claimed.test_id
ORDER BY claimed.duration DESC NULLS FIRST

• The pending-stats aggregate and the locked scan are served by the partial claim index (pipeline_id, job_group_id, duration DESC) WHERE node_index IS NULL; claimed rows leave it immediately so claim cost stays flat as history grows.
• Retry/replay is a read-only PK-prefix scan within one pruned partition.
• Seeding interns paths/groups per project with ON CONFLICT DO NOTHING and joins a VALUES list against the intern table (no id round-trip through Ruby), batched at 1,000 rows.
• The tables are brand new (empty in production — created in !248189 (merged)), so Database Lab plans are trivially empty; happy to attach postgres.ai plans on request once seeded.

Verification

• 34 request spec examples (spec/requests/api/ci/test_balancing_spec.rb): job token policies shared example, FF off → 404, PAT → 404, non-parallel → 422, guest job user → 403, seed/claim/retry/drained flows, cross-node dedup, params bounds, QueryRecorder N+1 guard.
• 23 service spec examples: budget taper, slowest-first, NULLS-first ordering, cross-node exclusivity, drain loop, idempotent seeding, group interning (parallel suffix stripped), pipeline/group isolation.
• 227 frontend jest examples for the token access UI pass with the new policy.
• gitlab:permissions:validate, RuboCop, ESLint, markdownlint, Vale all green; generated docs (fine_grained_permissions.md, granular token REST docs, GraphQL reference, OpenAPI tags, gitlab.pot) regenerated.

No changelog: everything is behind the default-off ci_parallel_test_balancing feature flag.

References

• Implements #607450 (closed)
• Schema MR: !248189 (merged)
• Parent: #606465