ci: cache taint analysis between gitlab-advanced-sast runs

What

Sets one variable on gitlab-advanced-sast:

gitlab-advanced-sast:
  variables:
    GITLAB_ADV_SAST_INCR_SCAN: "true"

Taint signatures for unchanged code are read from a cache instead of recomputed. The target set is untouched and every file is still scanned, so this is a pure cache with no coverage trade-off.

No fleet move, no tags: override, no runner minutes at a different rate.

Measured result

The job halves: 10.3 minutes against a 19.0-minute main median, a 46% reduction. Reproduced on two independent runs.

	Cold cache	Warm run 1	Warm run 2	main baseline
Cache lookup	292s	5s	5s	none
Scan	17.4 min	9.6 min	9.5 min	18.5 min
Job total	22.9 min	10.3 min	10.3 min	19.0 min (n=24)

Both halves improve: the lookup collapses from ~5 minutes to 5 seconds, and the cached signatures cut the scan by 45%. The two warm runs differ by 0.16% on scan wall time (573,267 ms and 572,369 ms), so this is a stable property rather than one lucky run.

The hit is unambiguous in the trace, naming the seeding job and its commit:

▶ Successfully downloaded ts-cache.sqlite.gz from job 16149226897
▶ Successfully decompressed cache file to /tmp/glas-cache/ts-cache.sqlite
▶ Successfully retrieved cache from commit 934df00366d9613074749823594f1ce671d8ceae
▶ Running incremental full scan

How this was measured

The cache cannot be exercised by ordinary means before merge: only a job carrying the variable writes one, and the analyzer looks for it along the default branch's history, where the variable does not exist yet. Four runs on this branch all reported No suitable cache found, over 396, 396, 392 and 225 commits. Rather than merge on faith, the cache was seeded deliberately and then removed.

1. Seed. A one-off pipeline on main with the variable passed as a pipeline variable, leaving main's committed configuration untouched:

   POST /projects/:id/pipeline   ref=main
        variables[][key]=GITLAB_ADV_SAST_INCR_SCAN  variables[][value]=true

   Pipeline 2796811059 (source api), job 16149226897, 23.3 min. It missed the cache itself, being the first, so its duration is not a result. Checked beforehand that main pipelines contain no release, deploy or publish stage jobs, only validate, so the trigger could not ship anything.

2. Verify the seed. ts-cache.sqlite.gz, 22.5 MB, inside that job's artifacts.zip, on commit 934df0036.

3. Build a valid reader. A fresh MR pipeline, so its merge ref contained the cache commit, confirmed with git merge-base --is-ancestor.

   A plain job retry was rejected as the method: the previous MR pipeline's merge ref was created at 13:03 and the cache commit landed at 13:27, so a retry would have missed for an unrelated reason and produced a false negative.

4. Measure. Jobs 16150119474 and 16150483245, both reading the seeded artifact.

5. Clean up. The seeded artifact was deleted from main's history afterwards, so no state from the experiment remains. Only the pipeline record stays, which needs Owner to remove and carries nothing.

The pass criterion was fixed before the runs: merge if the job comes in under 19.0 minutes with no findings lost, close the MR otherwise.

Consequence for this MR's own pipeline

With the seed deleted, pipelines on this branch are cold again and will show roughly 23 minutes. That is expected and is not evidence against the numbers above; the job traces cited by id remain. A real cache appears when the first main pipeline after merge writes one, and every run from then on reads it.

One operational caveat found the hard way

The lookup misses if the cache artifact is younger than roughly a minute. The first reader began searching 61 seconds after the seeding job uploaded its artifact and reported No suitable cache found in 262 commits; an identical retry against the same artifact, aged about 10 minutes, hit in 5 seconds. So a pipeline starting immediately behind the one that seeded the cache can still miss. This costs lookup time, not correctness.

Peak memory rose from 1,085 MiB to 1,549 MiB with the cache loaded, far below any limit this runner has.

Coverage did not regress

Configuration	Findings
main full scan (job 16145629187)	12
Cold cache (job 16147550685)	12
Warm run 1 (job 16150119474)	13
Warm run 2 (job 16150483245)	13

Nothing was lost: zero findings missing against either the cold run or main's full scan, and the two warm runs report identical sets. One extra Medium appeared, internal/format/maven/download.go:599 "Improper handling of highly compressed data".

That extra finding is not attributed with confidence. The file is unchanged since 2026-08-19 and all runs used identical analyzer v2.54.0 and rules v1.2.318, so version drift is ruled out. But 68 Go files changed on main between the full scan's commit and the warm runs', including dependency-injection wiring that cross-file analysis walks, so a newly reachable taint path is plausible; analyzer nondeterminism under its 30s rule timeout is the other candidate. Either way the deviation is in the safe direction, and the finding deserves triage on its own merits, independently of this MR.

Why, and what was ruled out

gitlab-advanced-sast is the most expensive job in the pipeline, and the scan is effectively all of it. It is also getting worse on its own: between 2026-08-21 and 2026-08-27 the in-scope tree grew 10.5% while the job's median grew 37%.

Date	In-scope Go files	Lines	Job median
2026-08-21	656	212,174	13.9 min (n=19)
2026-08-27	712	234,511	19.0 min (n=24)

Two other levers were measured on this branch and both produced nothing. Both are recorded so they are not retried.

Multi-core: no effect

ADVANCED_SAST_AVAILABLE_CPUS, ADVANCED_SAST_AVAILABLE_MEMORY and SAST_SCANNER_ALLOWED_CLI_OPTS: --multi-core 2 did take effect: job 16010150956 reports "MultiCore":2 and runs two workers. It bought nothing, 13.7 minutes against a 13.9-minute single-core median, because the workers split the rule set rather than the target set:

⏱️ Group 1 took 12m18.392302134s for 812 rules
⏱️ Group 2 took 13m5.864842156s for 1854 rules

Against 13m36s single-core. Wall time follows the slower worker, so 2.3x the rules cost 6% more time. Rule count is nearly free; per-file work is the cost.

Differential scanning: slower, and it lost a finding

ADVANCED_SAST_PARTIAL_SCAN: differential was measured across three runs and narrowed nothing. Worse, it made the scan slower than a full scan:

Configuration	Job	Scan	Files walked
Differential (job 16146341260)	27.8 min	20.9 min	1,186
Differential (job 16146397124)	28.3 min	21.5 min	1,186
Full scan (job 16147550685)	22.9 min	17.4 min	1,188

So removing it recovered roughly 3.5 to 4 minutes of scan time, and it held even though this MR's diff is a single YAML file with zero Go files, the best case the mode can be given.

The reason is structural: the whole-tree parse and the cross-file graph the analyzer needs in order to find "files that import the changed files" both happen regardless of the diff, and that is where the time goes. total_files stays at ~1,186 and lightz still runs with -deep_inter_file. The diff list only reaches what gets reported.

It also cost coverage: the differential run reported 11 findings where the full scans report 12, missing internal/format/npm/streampub/sink.go:3 [Low] on a file unchanged since 2026-08-20. A real coverage cost in exchange for negative time, so the mode is left off.

Both dead ends share one root cause, which is why this MR lands on a cache rather than a third scope variable: the dominant term is the whole-tree parse plus cross-file graph, and neither core count nor diff scope touches it.

Conflicts

.gitlab/ci/common-ci-tasks-patches.yml is also touched by !924 (Draft, last updated 2026-07-27), which removes the test:go:integration and post-test:go:coverage stanzas from the top of the file.

No conflict is expected. !924's hunk covers lines 1 to 19; this MR appends a new top-level key at the end. Whichever lands second applies cleanly.

An earlier revision of this description claimed !924 deletes the file and is semantically incompatible with this one. Both claims were wrong: its API diff reports deleted_file: false. It also cited !818 (closed), which has been closed since 2026-08-25.

Testing

CI plumbing with no Go surface, so no unit test applies. GET /ci/lint at 56fe0221f returns valid: true with no errors or warnings, and the merged gitlab-advanced-sast block carries the variable alongside the ts-cache.sqlite.gz artifact and its expire_in: 3 days that the template already provides.

Behavior verified end to end against real job traces, as recorded in Measured result and How this was measured.

Related to #751