## Why this exists Pipeline duration in this project is not a scheduling problem. It is a small number of slow tests. This item holds the findings and tracks the work as child items. All figures are measured, reproducible, and dated. The collectors live in https://gitlab.com/gitlab-org/ci-cd/package-stage/ai-automations and the full write up is `reports/ar_ci_test_findings_2026-08-27.md` there. ## Finding 1, pipeline level: there is no serialisation to remove 16,968 in scope pipelines, 2026-03-04 to 2026-08-25. 38,931 `duo_workflow` pipelines excluded as agent runs rather than CI. - **127 of about 139 jobs declare `needs: []`.** Almost nothing waits on anything. - The slowest single job is a median **1.00** of its own pipeline's duration, and above 0.9 in **94%** of 16,597 pipelines. - So pipeline duration is `max(job duration)`. Re-staging, reordering and adding parallelism cannot help. Only making an individual job faster can. Wall clock and runner time answer different questions and should not be mixed. Merge request median **10m58s** wall against **1.1h** runner, a 6.2x parallel factor. Main **14m30s** against **2.3h**, 9.4x. Wall clock is developer waiting time. Runner time is spend. Counterfactual over 11,871 merge request pipelines: halving `test:integration` moves the median to **6m55s**, a 37% saving. Removing it entirely reaches only 6m19s, 42%, because another job becomes the ceiling. **Halving is worth nearly as much as deleting**, so this does not have to be solved perfectly to be worth solving. Which job is the ceiling, share of merge request pipelines: `test:integration: [POSTGRES, 16]` 20.9%, `[POSTGRES, 18]` 20.0%, `go_unittests` 13.7%, `goreleaser_build` 9.7%, `gitlab-advanced-sast` 8.7%. Integration is the ceiling in roughly 60% of pipelines while running on only 50% of them, so returns stop sooner than a duration ranking suggests. ## Finding 2, job level: top 5 by median All five have n at or above 2,400, as of 2026-08-25. | Job | Median | p95 | Runs | Total runner time | |---|---|---|---|---| | `test:integration: [POSTGRES, 16]` | 14m30s | 21m06s | 9,229 | 2,212.5h | | `test:integration: [POSTGRES, 18]` | 14m29s | 20m59s | 9,229 | 2,198.8h | | `test:integration: [POSTGRES, 17]` | 13m16s | 19m32s | 9,229 | 2,050.0h | | `test:integration:s3-aws: [instance_auth]` | 12m17s | 16m54s | 2,400 | 505.8h | | `test:integration:s3-aws: [key_auth]` | 12m15s | 16m53s | 2,400 | 505.8h | Ranking by **total** runner time surfaces a different pair: `go_unittests` 1,244.1h (median 5m34s, 13,755 runs) and `goreleaser_build` 1,007.9h (median 4m48s, 13,865 runs). Those are cost levers with **zero** wall clock effect and belong on a separate track. Inside a job, from 40 sampled `test:integration` runs using trace section markers: setup **1m02s (7.7%)**, execution **12m04s (90.4%)**, teardown **26s (3.2%)**. There is no setup overhead worth reclaiming. The suite is the cost. Of the 88s outside execution, `restore_cache` plus `archive_cache` is 70s and appears on only 29 of 40 jobs. ## Finding 3, test level: a handful of tests, and a long flat tail From 40 job runs, 106,391 observations, as of 2026-08-24: **25,724 distinct tests**, 8,032 of them top level. **Median test median 0.00s. 97.9% run in under one second.** **14 tests are half** the summed test time; the top 10 are **46.4%**. | Test | Median | Min | Max | Runs | Pass/Fail | |---|---|---|---|---|---| | `internal/storage/driver/s3.TestS3DriverConformance` | 673.06 | 6.37 | 697.42 | 3 | 1/2 | | `internal/storage/driver/gcs.TestGCSDriverConformance` | 360.46 | 350.88 | 360.46 | 2 | 2/0 | | `internal/datastore/migrations.TestMigrations_UpDownUp` | 104.90 | 101.08 | 105.81 | 3 | 3/0 | | `internal/datastore.TestNamespaceEncryptionKeyStore_RotationReads_TombstoneArmIsIndexBacked` | 54.99 | 23.95 | 59.29 | 6 | 6/0 | | `internal/datastore.TestNamespaceEncryptionKeyStore_ListLiveKeys_ShortCircuitsTheTombstoneProbe` | 54.61 | 25.14 | 59.23 | 6 | 6/0 | | `internal/datastore.TestNamespaceEncryptionKeyStore_ListLiveKeys_PagesWithoutSortingTheLiveSet` | 53.12 | 25.51 | 55.77 | 6 | 6/0 | | `internal/datastore.TestNpmPackageStore_CascadeSoftDeleteNpmPackage` | 48.19 | 38.97 | 163.94 | 6 | 6/0 | | `cmd/artifact-registry.TestWireStorage_CloudCDNPresent` | 47.69 | 40.59 | 47.82 | 4 | 4/0 | | `cmd/artifact-registry.TestWireStorage_CloudCDNBoots` | 38.78 | 26.23 | 41.23 | 4 | 4/0 | | `cmd/artifact-registry.TestLifecycle_DatabaseIntegration` | 37.68 | 19.43 | 39.60 | 3 | 3/0 | **Three caveats change what this table means. Please read them before acting on it.** 1. **Row 1 is a failure, not a slow test.** It failed 2 of 3 observations, and its min is 6.37s against a 673.06s median. That is a test that hangs and is killed, not one doing 673s of work. Triage it as correctness first. 2. **Row 2 is thin.** Two observations are not a median. It may be genuinely slow; it needs more runs before that is asserted. 3. **Durations are not stable.** `CascadeSoftDeleteNpmPackage` ran 38.97s and 163.94s in two jobs of the same pipeline at the same commit. Never quote a median here without min and max. Clusters among the stable, passing rows: - **Migrations, about 170s.** `TestMigrations_UpDownUp` migrates up, down, then up again, so its cost grows with every migration ever added. It worsens monotonically forever unless bounded or moved off the per merge request path. The `TestWidenFormatEnumMigration_*` rows are the same shape. - **Namespace encryption key store, about 163s.** Three tests at 53s to 55s, 6 runs each, all passing. Mins near 24s suggest contention rather than work. - **Wire and lifecycle boot, about 124s.** Each stands up application wiring. Probably correct to test; the question is frequency, not duration. **A sum of medians is not a duration.** Tests run partly in parallel and the observed ratio of summed test time to job duration spans 0.01 to 1.84. Any presentation of one must show the other. ## Baseline benchmark | Measure | Baseline | As of | |---|---|---| | Merge request pipeline wall clock, median | 10m58s | 2026-08-25 | | `test:integration: [POSTGRES, 16]`, median | 14m30s (p95 21m06s, n=9,229) | 2026-08-25 | | `step_script` within that job, median | 12m04s | 2026-08-25 | | Sum of top level test medians | 3,174.31s over 8,032 tests | 2026-08-24 | | Top 10 share of summed test time | 46.4% | 2026-08-24 | | Slowest stable passing test | `TestMigrations_UpDownUp`, 104.90s | 2026-08-24 | Regenerate with `ar-ci-metrics` first, then `ar-test-timings`, which reads the former's cache. In the other order the test figures silently describe an older window. **Known limit of this baseline:** it samples 40 of 109,854 eligible jobs. The top 10 share was 27.5% at 8 jobs and 46.4% at 40, so the figure is still moving with sample size. Widen the sample before treating it as a target. ## Plan of execution One child item per finding, three steps each: 1. **Identify the test**, with measured median, min, max, run count, and share of summed suite time. A thin row is not a finding, it is a request for more runs. 2. **Write an issue** with plausible solutions and an implementation plan. Options, not decisions: the owning engineer chooses. 3. **Open a merge request and validate it** against the baseline, quoting the same measure before and after.