ci: cut pipeline and simulation wall time (needs graph, shallow clone, native regression, no lint lock, faster sim churn)

Son of Odinrequested to merge
odin/ci-pipeline-speed into develop

Ships the pipeline speedups (needs graph, shallow clone, native regression, no lint lock), plus the simulation speedups from !5148 (merged). It targets develop now that !5142 (merged) has merged, and it is independent of !5143.

!5148 (merged) was retargeted to this branch and merged into it, with its three commits kept as they were (rebased, now c4b3d4f1, 104b77b2 and 80209b2e). Its description has the full measurements. a6aac759 follows up on it so the standalone churn and churn-balance stages also schedule their churns and restore the interval.

!5151 (merged) (short jobs on small runners, and the native sim build) has since been folded into this branch as 27c69f00.

Measured impact

These numbers come from this MR's 4 pipelines before 27c69f00, so before the !5151 (merged) changes. They ran from 2026-09-29 21:00 to 2026-10-04 19:00 UTC, when Finland was capped at 4 concurrent jobs and harry/hermonie were not yet online. The comparison is the other 152 MR pipelines from the same window. Numbers for the full branch are pending its first pipeline (2914507911).

Pipeline, green runs only Before, p50 (p90) After, p50 (p90)
End to end 75.9 (174.3) min 55.4 (64.7) min: −27% p50, −63% p90
Gate jobs done 37.7 (131.6) 17.8 (33.8)
Sim starts 42.3 (134.5) 21.9 (33.8)
Job run time, p50 (min) Before After Change
test-simulation 44.6 31.6 −29%
test-regression 19.2 9.2 −52%
unit-tests 5.8 4.0 −32%
unit-tests-race-bifrost 5.1 3.3 −37%
lint 5.8 4.8 −17%
proto-check-formatting 3.5 1.1 −67%
proto-lint / proto-check-breaking 1.7 / 1.3 1.1 / 1.1 −31% / −14%
generate 3.2 2.8 −13%
small jobs (xmr-e2e-gates, semgrep, test-mocknet-scripts) 0.9–1.2 0.4–0.8 −34% to −59%

Caveats:

  • Queue time. With stage order a waiting job sits in created, which GitLab does not count as queue time. With needs: it sits in pending, which it does count. So per-job wait times are not comparable, and the end-to-end timestamps above are used instead.
  • Since harry and hermonie came online (2026-10-04 19:00 UTC) the sample is small: 9 other pipelines and 3 of ours, over a weekend.
    • When the sim passes on its first attempt, this MR's pipelines take the gate time plus 31–34 min of sim, which is 36.6–64.7 min end to end.
    • Two of those three pipelines retried a sim that failed on the new runners: one hung for 62 min on harry, and one hit load timeouts on hermonie. These are runner issues, not changes in this MR.

Why

!5142 (merged) made CI more reliable; this MR makes it faster, both in how the pipeline schedules jobs and inside the simulation itself. The numbers come from 2026-09-15..29: 7,900 jobs, 530 pipelines, and phase timings from about 20 traces.

A first-try green pipeline takes 65 min at p50 and 91 min at p90. About 27 min of that p50 is test-simulation waiting on stage order before it even starts. There is one commit per change.

Commit Measured cost today Expected after
proto-format-check in one container 12–16 min: one docker run per .proto file (159) ~1s of checking plus job setup
test-simulation diagnostics only on failure up to 3.6 min of after_script on every passing sim 0 on success
shallow clone (GIT_DEPTH: 20) full 941MB clone per job, 78s average (p90 125s), ~14% of runner time 17MB clone
needs: instead of stage order sim started 27 min in at p50 (117 at p90) sim starts when its checks finish, ~10 min
lint lock removed lock wait p75 25 min, p90 64 min; lint started 52–126 min into pipelines and was the last job to finish on this MR (+106 min) lint starts with the pipeline and takes about as long as unit-tests
sim: schedule churns (c4b3d4f1, a6aac759) Churn 3–9 min and BondYield-ChurnOut 10–14 min, waiting for a 365-block retry tick churn about 20 blocks after it is requested; ChurnOut 3–4 min (CI, hedwig: 11.7 → 3.9 min)
sim: L1s start during the image build, ZEC seeding batched (104b77b2, 80209b2e) DAG started +8.5 min into the job; ZEC seeding 1m51s DAG at +6.4 min; ZEC seeding 2s
test-regression without dind, PARALLELISM 10 16–24 min of dind image build before the first suite on Finland (3.6–7.9 min on hedwig), then 12–18 min of suites ~1.5 min build from the warm Go cache, then ~7.7 min of suites (measured locally, 184/184 pass)

Expected effect: p50 pipeline wall time falls from about 65 to about 45–50 min. The p90 falls further because govulncheck, the build jobs and the side jobs no longer hold back the sim. After this, the sim itself is the critical path. Its remaining internals (cold image builds) are for follow-up MRs.

Details

needs: graph.

  • Every protobuf and test job starts with the pipeline.
  • test-simulation and test-regression wait only for generate, the three proto jobs, lint and unit-tests (.long-test-gate).
    • Those jobs catch most pipelines that fail a check before the sim, so a bad branch still usually doesn't burn a sim run. That matters because sim failures rise with runner load.
    • Those jobs caught 116 of the 122 pipelines that failed a check before the sim on 09-15..29. In 33 of them, lint was the only failing check, so keeping lint in the gate saves about 26 sim-hours per two weeks.
    • For a while lint was out of the gate. Its global resource_group lock started it 52–126 min into each pipeline on 2026-09-29, and the sim waited with it (+86 min on this MR). The lock is now removed (below) and lint starts with the pipeline, so it is back in the gate.
    • The sim no longer waits for:
      • govulncheck (held it back 22 times)
      • hedwig-smoke, go-tss, test-mocknet-scripts, xmr-e2e-gates and semgrep
      • on develop, the build jobs (held it back 30 times)
  • The gate entries are optional: true. That covers deployment branches, where lint is left out, and some merge trains, where unit-tests is.
  • unit-tests-race-bifrost still waits for unit-tests, to keep the CPU separation its comment describes.
  • The build jobs keep their existing stage order and publication rules. build comes before long-test and uses when: always, so image publication waits for neither the sim nor regression. This MR doesn't change that; it only corrects the comment that claimed otherwise.

Shallow clone. Only two jobs read history, and both set GIT_DEPTH: "0":

  • lint: trunk-ci.sh diffs against the merge base with origin/develop.
  • fork.

secret_detection keeps the template's 50. No other script, Makefile target or Go test runs git over history. I grepped for git log, rev-list, describe, merge-base, diff and exec.Command("git").

Lint lock removed. resource_group: trunk-lint (7d5cf77b) serialized lint across every pipeline and runner.

  • On 2026-09-22..29, lint waited for the lock 4 min at p50, 25 at p75 and 64 at p90.
  • It did not prevent the trunk internal errors it was added for: all 20 that week happened with no other lint running.
  • A local test used separate checkout paths sharing one trunk cache:
    • 4 runs at once on a warm cache: all clean.
    • 3 runs at once on a cold cache: 1 hit lockf() failed ... Resource deadlock avoided, a tool-install race.
  • scripts/trunk-ci.sh now retries once on that error as well as on the internal-error banner. A retry reruns every linter, so it cannot hide a finding.
  • --jobs=4 (from !5142 (merged)) bounds each run's memory now that several can overlap.
  • A per-slot trunk cache was tried first and reverted: a cold slot took 24 min, and seeding one copied many GB.

Simulation (from !5148 (merged), plus a6aac759). Only test/simulation and the Makefile's test-simulation-cluster path change; no consensus code.

  • core.ScheduleChurn votes CHURNINTERVAL so the next churn fires 20 blocks out, instead of waiting for the next 365-block retry tick.
    • Churn schedules right before it unhalts churning, and BondYield-ChurnOut right after its leave.
    • ChurnBalance schedules before it clears HALTCHURNING.
    • BondYield-Cleanup restores the mocknet interval in the cluster run.
    • The standalone churn and churn-balance stages (suites.Churn, suites.ChurnBalance) restore it with features.NewChurnIntervalRestoreActor after the churn or migration is verified. Without that, a scheduled interval could exceed 1,000 blocks, which would push a later rotation past the actor's 10-minute timeout.
  • prepare-mocknet-cluster starts the 16 released-image L1 chains in the background while the simtest and thornode images build. It is a shell background job, so it works with macOS make 3.81.
  • ZEC seeding plans each user's transfers once, broadcasts them, and waits once per user.
  • In CI on hedwig, the sim DAG went from 25.6–27.9 min to 19.6 min, compared with same-day sims that didn't have the change. A local same-machine A/B went from 24.9 to 17.3 min.

test-regression in the job container. It no longer builds ci/Dockerfile.regtest in an empty dind daemon.

  • It runs make _build-test-regression and make _test-regression in the golang image, reusing /go-build-cache and /go-mod-cache.
  • REGTEST_BIN_DIR, REGTEST_HOME_BASE and REGTEST_MNT_DIR point the harness at the checkout. make test-regression is unchanged for local use.
  • PARALLELISM goes from 6 to 10. Each suite file gets its own ports and home directory, offset by file index, so the worker count cannot collide.
  • Local timings with go1.26.5, all 184 suites passing and no export drift:
    • P=6: 760s
    • P=10: 462s
    • P=10 pinned to 8 CPUs: 469s The suite is bound by block time, not CPU.
  • GO_PRIVATE_AUTH_REQUIRED is dropped. With a warm module cache the job behaves like unit-tests, so a missing tss-lib-private credential only warns.

Measured so far (2026-09-29, before the lint-gate and regression commits)

Per job, against 11 other pipelines on the same runners the same afternoon:

  • get_sources: 34–302s (usually 55–130) → 3–10s
  • proto-check-formatting: 11–20 min → 0.5–4.9 min
  • The sim start did not improve (+86 min against +61 to +131), because it was waiting for lint. That is what the lint-gate commit fixes.

No pipeline had finished cleanly: Finland was running 9 sims at once and its disk filled up that afternoon.

Risks to watch

  • ZEC seeding margin. At PARALLELISM=12, ZEC would have 240 of its ~300 coinbase UTXOs in flight at once. CI runs 4.
  • Earlier L1 start. The L1s now start about 4 min before thornode, so they begin at greater heights. Full sims have passed with this.
  • CHURNINTERVAL votes. CHURNINTERVAL needs a supermajority of active nodes. If the votes land late, the churn falls back to the next retry tick, which is the old timing. A stage that schedules a churn must restore the interval; the suite test covers the standalone stages.
  • Regression ports. They now live in the job container's network namespace instead of dind's. That is safe with the docker executor's per-job networking (confirmed on hedwig). If a runner used network_mode = "host", two concurrent regression jobs would collide.
  • Peak CPU. In the first ~10 min, lint, unit-tests, generate and the proto jobs now overlap; before, the protobuf jobs ran first and the test jobs after them. Total runner time goes down, but the start of the pipeline is denser. The proto jobs are now mostly setup, so the added overlap is small. If unit-tests shows localhost timeouts again, that is where to look.

Considered and left out

  • Merging the proto jobs into one dind-free job in the proto-builder image. An image: pinned to the registry mirror breaks the MR that bumps protoVer, which is why 8758c44f was reverted. Pinning ghcr.io instead brings back its pull failures. With the formatting fix these jobs take ~3 min each and are off the critical path.
  • GIT_STRATEGY: fetch. It would skip even the shallow clone, but it reuses working copies that dind jobs write into as root. The shallow clone gets most of the gain without that risk.
  • Dropping the duplicate test-mocknet-startup from make test-simulation-cluster. It saves 38s but would change what the local target checks.

Validation

  • CI lint API with a dry run on develop: valid, no warnings. The merged YAML shows the intended needs and GIT_DEPTH on every job.
  • make proto-format-check: passes on the tree in 3s total. With a deliberately misformatted temp .proto it exits 1.
  • trunk check on the changed files: no issues.
  • The first five CI commits were built by stripping each change from the final file; stripping all five reproduces the base .gitlab-ci.yml byte for byte.
  • A suite test checks that each standalone churn and churn-balance stage restores CHURNINTERVAL exactly once, and only after its churn or migration verification.

CI verification (2026-09-29)

Pipeline 2894538135 is green; govulncheck is still allow_failure until !5137 (merged).

before this MR
proto-check-formatting 12-16 min (28.0 min on !5148 (merged) in the same window) 2.4 min
get_sources per shallow job 78s mean 3-13s (sim: 6s)
sim start stage order: +27 min p50 2s after its last gate job (lint)
sim after_script on success up to 3.6 min 2s (skipping failure diagnostics)
  • The sim passed (51.6 min, Finland) and so did regression (16.1 min, hedwig).
  • Pipeline wall time was not measurable. Jobs queued for about 35 min while the Finland runner was being repaired. The first lint was also OOM-killed on hedwig's 24g job limit. The OOM is a runner issue; the retry passed.
  • The biggest remaining pre-sim delay was the trunk-lint lock. Later commits removed it (see "Lint lock removed").

CI verification (2026-09-30, lint lock removed)

Pipeline 2895477249 on 7a1f54d3 is green. It does not include the sim commits yet.

  • Pipeline wall time: 66.9 min. For comparison, the previous run of this MR took 106.3 min, and other MRs' first-try-green pipelines take 65 min at p50.
    • It waited 12.7 min for a free runner before the first job; the baseline p50 wait is 0.1 min.
    • The sim was the last job to finish. In the previous run, lint (behind the lock) had been the last.
  • Lint started +18 min, down from +104. It hit a Trunk Internal Error; the retry passed, and the job took 3.3 min in total on hedwig's warm trunk cache.
  • The sim started 6.9 min after the first job (baseline p50 17.1).

Per job, against other MRs' passing jobs over 09-22..29 (p50):

before this MR
test-regression 28.3 min (hedwig 16.7) 9.2
proto-check-formatting 7.7 0.6
unit-tests 7.4 2.8
generate 5.1 1.5
proto-lint / proto-check-breaking 3.1 / 3.2 0.5 / 0.5
test-simulation 47.4 (Finland 48.5) 47.3 (Finland; without the sim commits)

🤖 Generated with Claude Code

Edited by Son of Odin

Merge request reports