Add models and tables for parallel test balancing
What does this MR do and why?
Creates the storage schema for Parallel Test Balancing — a duration-aware queue that distributes a CI job's parallel: test files across nodes so every node finishes at roughly the same time, and replays the same split on job retry. Builds on the existing parallel: keyword (CI_NODE_INDEX / CI_NODE_TOTAL). The claim/seed/retry API comes in a follow-up (see references).
This is an experimental feature which I aim to test on gitlab-org/gitlab pipelines. Initial results in the PoC show a reduction of 20-30 minutes from our pipeline wall clock time.
| Table | Purpose |
|---|---|
ci_test_balancing_test_splits |
Test splits (currently test file paths), per project. Append-only get-or-create; ids never recycled. Cleaned up on project deletion via LFK. |
ci_test_balancing_job_groups |
Job group names (from the arbitrary CI job name), per project. Append-only; LFK cleanup. |
ci_test_balancing_assignments |
Durable, write-once record of claimed test splits. One row per (pipeline, job group, test split), stamped with the pipeline's start time as partition key and the claiming parallel node in node_index. Daily RANGE partitions on pipeline_created_at, dropped after 30 days by the partition manager. |
The pending work queue itself lives in Redis (introduced in the API follow-up). ci_test_balancing_assignments is only the durable record of what each node was handed: every row is a claim (node_index is NOT NULL), written once and thereafter read-only history, replayed to give a retried node the exact same set of test splits.
The
test splitnaming (instead oftest) is deliberately granularity-agnostic: today a split is a whole test file path, but the name leaves room to split finer in the future without implying an individual test case.
Notes
ci_test_balancing_test_splitsandci_test_balancing_job_groupsaren't strictly necessary but these are repeated a lot since a project runs the same set of tests / jobs many times. So I decided to normalize them into their own tables instead of just storing them as strings in the assignments table.ci_test_balancing_assignmentsis partitioned bypipeline_created_atso that we can leverage PG partition pruning. All the jobs for a pipeline will have the samepipeline_created_atvalue so we only scan one partition.- Redis owns the transient pending queue; PostgreSQL is the durable system of record for claims so that individual job retries can run the same set of tests as the previous job with the same
CI_NODE_INDEX. We don't need to store this indefinitely since we don't need to support retries for very old pipelines. I just started with a 30 day retention policy but we can also lower this in the future. - I asked Claude to estimate how much storage this would use with our list of tests. For 21k tests and 1,000 pipelines per day, it estimates it at 85 GB / 30 days. Today we run ~900 pipelines per day and not all of them are tier 3, so I expect actual numbers to be lower.
References
- Related to #607420 (closed)
- Parent: #606465
- Follow-up (API): !249227 (merged)