## Summary The MR pipeline's median wall clock is 14.9 minutes. Three jobs account for almost all of it, and each is slow for a reason that is cheap to fix: `gitlab-advanced-sast` scans on a single CPU core, `docs:links` re-checks every external URL in the repository with no cache, and `test:integration` spends 96% of its run inside one strictly serial package. Fixing all three is expected to take non-integration pipelines from about 12 minutes to about 6, and integration pipelines from about 17.5 minutes to about 12.6. ## How this was measured Job timings come from the REST API over the 20 most recent `merge_request_event` pipelines as of 2026-08-20, of which 8 succeeded (2776056990, 2776053425, 2776040384, 2776028427, 2776027763, 2776022787, 2776006754, 2776004578). Per-job breakdowns come from the job traces of pipeline 2776028427. Anyone re-running this should expect the absolute numbers to drift as the suite grows; the ratios are the durable part. ## Baseline Median successful MR pipeline: 14.9 minutes (n=8, range 10.9 to 18.1). The two pipeline shapes differ in size as well as in duration, so a single job-count figure does not describe both: | Shape | Jobs | Job-minutes | Longest job | Median | |---|---|---|---|---| | Integration tests not triggered | 30 to 31 | 36 to 44 | `gitlab-advanced-sast` | 12.6 min | | Integration tests triggered | up to 70 | up to 201 | `test:integration: [POSTGRES, 18]` | 17.3 min | The job-count and job-minute columns are the observed range across the 8 samples; the fullest pipeline (2776028427) is 70 jobs, of which 68 report a duration, 71 counting retries, and 201.2 job-minutes. Every locally defined job carries `needs: []`, so the validate stage is a flat fan-out and **the critical path is the single longest validate job**. Stages otherwise serialize, and a full pipeline also has jobs in `.pre`, `application-security-testing`, `release`, and `deploy`; those do not extend wall clock only because the `release` and `deploy` ones are manual. A future non-manual job in a later stage breaks the premise this whole issue rests on. ## Finding 1: `gitlab-advanced-sast` scans on one CPU core Median 12.6 minutes across all 8 samples. It was the last job to finish in 4 of them and in the top 3 in 6 of them. From its own trace: ``` Detected 2 CPU Cores ... Using 1 cores for multi-core ... Running full scan Scan finished wall_ms=816408 user_ms=802473 peak_memory_mib=1088 ``` 802s of user time against 816s of wall time means 98% of the job is one single-threaded scan over 1059 files. Per the [Advanced SAST docs](https://docs.gitlab.com/user/application_security/sast/gitlab_advanced_sast/), multi-core is on by default, the analyzer reads the runner tag to look up core and memory counts, and it wants 4 GB of memory per core. The default fleet gives it 2 cores and 8 GiB. `saas-linux-large-amd64` is 8 vCPU and 32 GB, which is exactly 4 GB per core. **Try the cheap experiment first.** The trace says the analyzer detected 2 cores and used 1. The job's `tag_list` is empty (verified on job 2776028427, which ran on `green-5.private.runners-manager.gitlab.com`), so the tag lookup the docs describe has nothing to read and the analyzer cannot confirm it has the 4 GB per core it wants. Getting it to use the second core it already sees is a smaller change than a fleet move and would establish how much of the win is parallelism before any runner minutes are spent differently. If that is not enough, append a `tags:` override to `.gitlab/ci/common-ci-tasks-patches.yml`, which already exists for overriding component-provided jobs and is included last so the override merges on top: ```yaml gitlab-advanced-sast: tags: - saas-linux-large-amd64 ``` Expected 12.6 minutes to roughly 3 to 5. **This is not cost-neutral, and the earlier claim that it was does not hold.** The job currently runs on a `gitlab-org` private runner fleet rather than the SaaS shared fleet. If that fleet is not billed at the SaaS per-minute rate, moving to `saas-linux-large-amd64` at a 3x factor is new spend rather than a wash, and the shorter run does not offset a rate the job is not currently paying. Price it against the private fleet's actual billing before quoting a cost figure. Separately, someone should confirm that moving a security scanner across fleets is acceptable. ## Finding 2: `docs:links` re-checks every external URL, uncached, on every pipeline Median 6.6 minutes, the second-longest job whenever integration tests do not run. Defined at `.gitlab/ci/docs.gitlab-ci.yml:38-46`. From its trace: ``` $ lychee '**/*.md' 6202 Total (in 5m 22s) | 1957 Unique | 6175 OK | 0 Errors | 27 Excluded | 30 Redirects ``` That is 5m22s of pure network I/O. The job has no `changes:` rule, so it runs on Go-only MRs too, it caches nothing between runs, and it is `allow_failure: true`, so it cannot fail the pipeline. It only holds it open. Fix: ```yaml docs:links: script: - lychee --cache '**/*.md' cache: key: lychee-links paths: - .lycheecache # allow_failure does not make the job's exit status zero, and cache:when # defaults to on_success. Without `always`, any pipeline with one dead link # skips the cache upload and the next run pays the full 5m22s again. when: always ``` Expected 6.6 minutes to well under 1. Link rot is caused by the internet rather than by any given MR, so a one-day cache costs no real coverage. Two details behind that block: - `--max-cache-age` already defaults to `1d`, so passing it changes nothing. Pass it only if the intent is to pin the value against a future default change, and say so if you do. - lychee caches error statuses alongside successes, so a transient 5xx sticks for the cache lifetime. That is acceptable here precisely because the job is `allow_failure: true` and cannot block anything, but it is the reason not to make this job blocking later without revisiting the cache. On the deferred question of adding `changes: ["**/*.md"]`: with the cache in place the job runs in under a minute, so the rule buys little and costs the per-pipeline check that catches rot on the days nobody touches a Markdown file. Recommendation is to keep the job running everywhere and skip the rule. ## Finding 3: `test:integration` is 96% one strictly serial package Median 17.3 minutes on the PG 18 leg, which sets the critical path whenever integration tests run. Parsing the trace per package: | | | |---|---| | Whole run | `DONE 17637 tests, 31 skipped in 858.853s` | | `internal/datastore` | spans 30s to 851s, 1296 top-level tests | | Every other package | finished by 313s | | Sum of the datastore tests' own durations | 821s, against an 817s wall span | That last row is the finding: **1.01x parallelism**. The package runs strictly serially for about 9 minutes while 3 of the runner's 4 cores sit idle. ### What is actually in that binary The package's test surface is 204 top-level `*_test.go` files, and `go test -tags=integration` compiles all of them, because the untagged files carry no build constraint. The split matters for every decision below: | Files | Count | Call `t.Parallel()` | |---|---|---| | Carrying `//go:build integration` | 119 | 2 | | Untagged, compiled into both the unit and integration binaries | 85 | 84 | The 119 tagged files hold 749 top-level `Test` functions; the remaining 544 of the 1293 the source yields come from the untagged files. `internal/datastore/schemas` carries no integration-tagged file at all, so it contributes nothing to this job. The serial behavior is deliberate for the tagged half: `internal/datastore/postgres_integration_test.go:3` carries `//nolint:paralleltest // integration tests share a database`. Adding `t.Parallel()` there is a refactor of the test harness, not a quick fix, and is out of scope here. Note that 84 of the 85 untagged files *do* call `t.Parallel()` and run inside this same binary, and the package still measures 1.01x. So the serial tail is the tagged suite specifically, and the parallel unit tests sharing the binary are too small a fraction of the 821s to show up. ### Sharding across CI legs Sharding avoids the harness refactor entirely, because each leg gets its own Postgres service and `TestMain` already calls `testutil.SetupIsolatedTestDB`. No test code changes and no shared-database risk. Pull the top-level `./internal/datastore` package into its own job and shard it: ```sh # Shard membership comes from source, not from `go test -list`. TestMain at # postgres_integration_test.go:23 provisions and migrates a database before # m.Run() reaches the -list flag, so listing costs a full setup per shard and # prints nothing at all when Postgres is absent. # # All *_test.go files, not just the integration-tagged ones: the untagged files # compile into this binary too, and filtering them out of the -run regex would # stop them running here. tests=$(git grep -hoE '^func Test[A-Za-z0-9_]+' -- ':(glob)internal/datastore/*_test.go' | sed 's/.*func //' | sort -u) run=$(printf '%s\n' "$tests" | awk -v s="$SHARD" -v n="$SHARD_TOTAL" 'NR % n == s - 1' | paste -sd'|' -) go tool gotestsum --format testname -- -tags=integration -count=1 -timeout 1380s \ -coverprofile=coverage/integration.out -run "^($run)$" ./internal/datastore ``` Measured on `main` at the time of filing: 1293 names, splitting into 4 shards of 322 or 323, each a `-run` regex of roughly 17 KB. Well inside `ARG_MAX`, and the alternation has no `/` in it, so subtests of a matched test all run. That 1293 is three short of the trace's 1296, which is close enough to trust the approach and not close enough to leave unexplained: reconcile the three before relying on the split, since a name the regex never matches is a test that silently stops running. There is precedent for the split: `internal/datastore/migrations` was carved out of this same job for this same reason, and the comment at `.gitlab-ci.yml:911` records why. ### Implementation notes Four of these are load-bearing. A shard series that skips the first two ships green and saves nothing. - **Exclude the top-level package from the residual job.** Its filter is `grep -vE '/datastore/(jet|migrations)(/|$)'`, which does not match `.../internal/datastore` itself. Unless that pattern gains `/datastore$`, the suite runs in both the shards and the old job, the 9 minutes stay on the critical path, and the shards' speed makes it look fixed. - **Decide what happens to coverage, explicitly.** The job carries `-coverprofile`, a cobertura conversion, and `coverage: *coverage-regex`. Dropping them from the shards removes the largest integration suite from the coverage report; keeping them per shard makes each shard report a partial total, and GitLab's pipeline coverage is the average of the per-job values, so the reported number falls. Either is defensible. Pick one in the MR description rather than by omission. - **Reuse the existing anchors.** `rules: *integration-rules`, `services: [*pg-test-service]`, `artifacts: *cobertura-report`, and `coverage: *coverage-regex` are all anchored on `test:integration`, and the migrations job is the worked example. A new job without the rules anchor never triggers on an MR at all. - **`.pg-version-matrix` cannot be extended here.** `extends` keeps only the last `parallel:` key, so combining `PG_VERSION` with a shard index means writing the matrix inline. `CI_NODE_INDEX` and `CI_NODE_TOTAL` are set only under `parallel: N`, not under `parallel: matrix`, so the shard index has to be an explicit matrix variable either way. - **Price the added job-minutes.** `.pg-version-matrix` is three versions (16, 17, 18), so 4 shards is 12 jobs replacing 3. The migrations job's own comment budgets about 2m of setup and 1m of coverage tail per job, and every shard provisions and migrates its own database. That is on the order of 30 or more job-minutes added for the wall-clock win. The win is still the point; the cost should be stated rather than left for a reviewer to notice. - **The win is capped.** Once `test:integration` drops to roughly 7 minutes, `test:integration:migrations` (12.6 min) becomes the new ceiling for integration pipelines, so the realized gain is about 17.5 minutes to 12.6. Speeding up the migrations suite is separate follow-up work, and no issue tracks it yet; whoever picks up MR 3 should file one so the cap has a home. ## Bonus: no `interruptible` anywhere Neither `.gitlab-ci.yml` nor any file under `.gitlab/ci/` sets `interruptible:`, and there is no `workflow: auto_cancel:`. Superseded pipelines therefore run to completion. What that costs depends on the shape: up to about 201 job-minutes when integration tests are triggered, but only 36 to 44 when they are not. Three MRs in the sampled window pushed twice inside 15 minutes; the compute recovered depends on which shape those pipelines were, so quote the shape alongside any figure. `default: {interruptible: true}` recovers that compute. Note that it will not shorten any single pipeline: queue times in the sample were 3 to 37 seconds, so this is a cost and fleet-contention win rather than a latency one. Worth folding into one of the MRs below rather than carrying its own. ## Proposed MRs Three independent MRs. They touch disjoint files, so they can run in parallel rather than as a stack: | MR | File | Change | Expected effect | |---|---|---|---| | 1 | `.gitlab/ci/common-ci-tasks-patches.yml` | SAST cores, then a runner tag if needed | 12.6 min to ~4 min | | 2 | `.gitlab/ci/docs.gitlab-ci.yml` | lychee cache with `when: always` | 6.6 min to <1 min | | 3 | `.gitlab-ci.yml` | split and shard `internal/datastore` | 17.3 min to ~7 min | No plan file is proposed. Every one of the 36 plans under `docs/plans/` is a spec-driven feature slice, `docs/dev/labels.md` classifies CI plumbing as upkeep, and prior CI work such as #389 shipped without one, though as a single-line timeout change rather than a three-MR series. If the team reads a three-MR CI series as an initiative under the plan-MR guardrail, say so on this issue and a plan MR goes first. Each MR should quote its before and after numbers from its own pipeline, since the point of the change is the number. ## Before opening MRs 1 and 3 Two drafts targeted the same files when this issue was filed on 2026-08-20: - !818, which renames `test:integration` to `test:go:integration`, last updated 2026-07-06. - !924, which deletes `.gitlab/ci/common-ci-tasks-patches.yml`, last updated 2026-07-27 and carrying merge conflicts. Both were draft on 2026-08-20 and neither had moved in weeks, but both overlap MRs 1 and 3 above. Whoever opens those two re-runs the search that guardrail 23 in `CLAUDE.md` describes and checks whether either draft has since progressed. If one is still open and still touches the file, ask the operator before pushing. While !924 is open, MR 1 adds a `tags:` block to a file that MR proposes to delete, so whichever lands second has to rebase onto the other's outcome. ## Corrections after review, 2026-08-20 Re-verified against `main` and the REST API after filing. The medians all reproduce exactly (SAST 12.6 at n=8, `docs:links` 6.6 at n=8, `test:integration` PG 18 17.3 and migrations 12.6 at n=4 each), as do the flat fan-out, the 1.01x parallelism, the `//nolint:paralleltest` line, the `docs:links` definition, the absence of `interruptible:`, the 36 plans, and the state of !818 and !924. What changed: - The `t.Parallel()` count was wrong in both numbers. "5 of 122 integration-tagged files" counted three unit-test files that only mention `//go:build integration` in a comment explaining why they sit outside the suite. The anchored count that `scripts/ci/check-integration-test-wiring.sh` uses gives 119 tagged files, of which 2 call `t.Parallel()`. The correction strengthens the finding. - The proposed shard snippet used `go test -list` to enumerate tests. That was verified to fail: `TestMain` provisions a database before `m.Run()` handles the flag, so the listing prints nothing without Postgres and costs a full setup with it. Replaced with a source-derived list. - The snippet also enumerated the package non-recursively without excluding the top-level package from the residual job's `go list` filter, which would have run the suite twice, and it dropped `-coverprofile` silently. - MR 1's cost-neutrality claim was removed. It assumed both fleets are billed at the same base rate, which the issue's own observation about the private fleet contradicts. - MR 2's cache needed `when: always`, since `allow_failure: true` does not make the job's exit status zero and `cache:when` defaults to `on_success`. - "73 jobs and roughly 206 job-minutes" overstated the observed maximum, and applied one shape's figure to both. Replaced with the per-shape ranges. - The guardrail number for the issue-and-MR search is 23, not 22.