ci: remove infra and external CI failures (retries, tss-lib-private credential, image mirrors, tools image)

Summary

Removes CI failures that don't come from the branch being tested. The data comes from a 2-week analysis of THORNode CI (2026-09-14..28, 7,389 jobs, 1,062 blocking failed attempts, all on the Finland runner). There is one commit per fix. It also stops superseded and already-failed MR pipelines from holding runners (see Auto-cancel below). Adding runner capacity, test-simulation flakiness and govulncheck are out of scope.

This MR targets about 230 of the 1,062 blocking failures (~22%). The follow-up MR (tools image) adds about 38 more.

Commit	Failures (2 weeks)	Expected after merge
Retry trunk on "Trunk Internal Error"	21 lint	~0
Retry infrastructure failures	73 (24 source fetches, 36 runner_system_failure, 9 no_updates_running, 4 dead dind)	most retried automatically
tss-lib-private credential	62 (30 test-regression + 32 test-simulation)	0 once admin step 1 below is done; until then, a clear fail-fast message
In-dind pulls off ghcr.io / Docker Hub	55 proto-builder (ghcr.io) + 19 Docker Hub logins + 2 ubuntu base pulls	~0 once develop has run mirror-ci-images
Tools image (build job only)	38 apk add mirror errors	fixed by the follow-up MR

Two corrections to the original analysis:

• 14 of the 41 "git fetch" failures were traces that hit a rate limit while being downloaded, not real failures.
• 32 test-simulation failures filed under "registry/mirror/proxy" were really tss-lib-private access failures.

Changes

Trunk retry (scripts/trunk-ci.sh). Trunk is retried once, and only when it exits non-zero with the Trunk Internal Error banner. Real findings still fail on the first run. The exit status is kept. Tested against a fake trunk: ok / ie→ok / ie→ie / findings / ie→findings.

Trunk parallelism (scripts/trunk-ci.sh). -j8 becomes --jobs=4. On full-repo runs (develop, merge trains, MRs touching go.mod/.trunk), 8 golangci-lint processes at ~3.2 GiB each filled hedwig's 24 GiB job container and golangci-lint exited -9 (develop lint job 16821715068). Four at once peaks near 13 GiB.

Infrastructure retries.

• New GET_SOURCES_ATTEMPTS, ARTIFACT_DOWNLOAD_ATTEMPTS, RESTORE_CACHE_ATTEMPTS and EXECUTOR_JOB_SECTION_ATTEMPTS (3 each). Source-fetch failures (fetch-pack: unexpected disconnect, GitLab is currently unable to handle this request due to load, RPC failed; curl 92) all happen in get_sources. GitLab reports them as script_failure, so only this variable can retry them.
• New default: retry (max 2) on infrastructure reasons only, never script_failure. It uses the GitLab 19.1 reasons: runner_external_dependency_failure, runner_interrupted, stuck_pending_with_matching_runners, no_updates_running, plus runner_system_failure, api_failure, scheduler_failure, unknown_failure and data_integrity_failure.
  • stuck_or_timeout_failure is deprecated; CI lint warns about it and new jobs never report it.
  • Server-side execution timeouts are deliberately not retried.
• 34 of the 36 runner_system_failure were the runner failing to pull docker:28.5.2(-dind), golang, alpine or semgrep from Docker Hub.
• .wait-for-dind waits up to 60s for the daemon, then exits 75. retry:exit_codes: [75] treats that as an infrastructure failure. It replaces build-thornode's 30s loop.
• buf_retry, test-simulation and test-regression set their own retry:, which replaces default entirely, so they now list the same reasons.

tss-lib-private. Every test-regression and test-simulation job started by codehans, boonew, kukulkan04 or GrassRootsCrypto failed with no secure protocol found for repository. CI_JOB_TOKEN has the permissions of the pipeline user, and those users can't read tss-lib-private. The build secret was present; "no secure protocol" is just the error go gives once every scheme was refused. Their unit-tests pass only because the runner's shared /go-mod-cache already holds the module.

• scripts/ci-go-private-auth.sh checks each credential with git ls-remote and uses the first that works: an optional TSS_LIB_PRIVATE_READ_TOKEN, then CI_JOB_TOKEN. The recommended fix needs no token (admin step 1).
• It configures git for gitlab.com/thorchain/tss/ only, and exports GO_PRIVATE_GIT_CREDENTIAL. The Makefile and docker-compose forward that as the go_private_auth BuildKit secret, falling back to CI_JOB_TOKEN.
• dind build jobs set GO_PRIVATE_AUTH_REQUIRED=1. They stop right away with a message naming the pipeline user (exit 75 and a retry if gitlab.com was unreachable). Other jobs only warn.
• Locally, make test-regression GO_PRIVATE_GIT_CREDENTIAL=user:<PAT> now works without any SSH workaround.
• Fixed ci/Dockerfile.simtest: its first go mod download left /root/.gitconfig, with the credential in it, in an image layer.
• go-tss now excludes fork MRs, which keeps honest forks away from the credential. This is not a security boundary. ci_allow_fork_pipelines_to_run_in_parent_project is on, and a fork MR pipeline that a maintainer runs in this project uses the fork's .gitlab-ci.yml and receives every unprotected project variable. The only real boundary is the maintainer who reviews the fork's diff before clicking "Run pipeline".

In-dind pulls.

• mirror-ci-images, on protected branches, copies ghcr.io/cosmos/proto-builder:$protoVer@$protoDigest into $CI_REGISTRY_IMAGE/ci/proto-builder with docker buildx imagetools create (registry to registry, no dind). The image digest is identical.
  • Pinned by digest. Running on protected branches only is not a registry ACL: any Developer's CI_JOB_TOKEN can push to this registry, and the project has no container repository or tag protection rules. So the Makefile pins the upstream index digest (protoDigest), CI sets only the repository (PROTO_BUILDER_REPO), and a replaced tag fails to pull instead of running. mirror-ci-images re-points a tag that has drifted from the pin.
  • The proto jobs and generate use the mirror, and fall back to ghcr.io (same digest) when the mirror doesn't have the pinned version.
  • The project registry is public, so every user and fork can pull it.
• Docker Hub docker login is retried 3 times, then skipped. It only raises the rate limit, and 19 jobs died on it. The Docker Hub and dependency-proxy setup was copied in four jobs; it's now one .dind-registries.
• debian-slim (thornode, events), ubuntu (monero mocknet) and openapi-generator get the DOCKER_MIRROR prefix. I checked that the dependency proxy serves the tag@digest pins.
• test-simulation no longer downloads docker-compose v2.28.1 from GitHub on every run. It uses the v2.40.3 plugin that ships in docker:28.5.2. docker compose config resolves the mocknet and mocknet-cluster profiles identically under both versions.
• Job and service images stay on Docker Hub, on purpose. The four users above can't use the group dependency proxy either: every one of their pipelines logs Dependency proxy missing golang:1.26.4; falling back to Docker Hub. A registry mirror pinned in image: breaks any MR that bumps its tag, which is why 8758c44f was reverted. The retries above cover those pulls instead.

Tools image.

• ci/Dockerfile.ci-tools is docker:28.5.2 plus bash, curl, findutils, git, jq, make, protoc, python3, sed, wget and Go 1.26.4.
• build-ci-tools-image builds it on a push to develop that changes the Dockerfile (or manually from a web pipeline on a protected branch), so promotions to stagenet/mainnet and API pipelines don't re-run it. It smoke-tests the image, pushes ci/tools:$CI_TOOLS_VERSION, and prints the pushed digest. It never overwrites a tag, so a tag pushed first by someone else makes it fail visibly.
• No job uses it yet (bootstrap). The follow-up MR odin/ci-tools-image-adopt switches the dind jobs and test-mocknet-scripts to it once develop has published ci/tools:1. It must reference ci/tools by the digest build-ci-tools-image printed, not by :1, because build-thornode holds the release-image push credentials.

BUF_PROXY on non-Finland runners. BUF_PROXY (a CI/CD variable) exists because buf.build returns 403 to Finland's egress IP, but that proxy only works from Finland. On the new hedwig group runner, every buf job failed with the server hosted at that remote is unavailable (buf's error for a refusing proxy). .buf-egress probes api.buf.build from the job and keeps BUF_PROXY only on 403 or no answer. Finland keeps its proxy, and other runners go direct. wget exits non-zero on 403, 404 and no answer alike, so the probe discards its exit status (job scripts run with errexit and pipefail) and decides on the HTTP status line alone. Tested in docker:28.5.2 under set -eo pipefail with busybox and GNU wget for direct 404/403/unreachable/unset.

Auto-cancel. No job set interruptible. The project's "Auto-cancel redundant pipelines" setting (enabled) skips any pipeline that has a started non-interruptible job, so in practice it never fired: every push to an MR ran the full pipeline, including test-simulation and test-regression, to completion. Jobs are now interruptible: true by default, and workflow:auto_cancel sets on_new_commit: interruptible and on_job_failure: all. A new push cancels the older MR pipeline, and the first failed job cancels the rest of its pipeline. GitLab doesn't count a failure that will be auto-retried (Ci::Build#can_auto_cancel_pipeline_on_job_failure? → !auto_retry_expected?) or an allow_failure job (govulncheck, mirror-ci-images), so the retries above still work. Tag and protected-branch pipelines publish images and docs, so they set both to none through workflow:rules:auto_cancel. Trade-off: a failing MR pipeline now reports only the first failure. Later jobs show as canceled rather than giving their own result.

Required admin steps (maintainer)

1. tss-lib-private access (recommended, no shared credential). Give codehans, boonew, kukulkan04 and GrassRootsCrypto Reporter on thorchain/tss/tss-lib-private. CI_JOB_TOKEN then works for their pipelines with no further change. Access is least-privilege, per-user, revocable and audited, and it ends with each job.

   Only if a shared token is still required: create a read_repository deploy token on tss-lib-private with a short expiry (e.g. 30 days) and scheduled rotation. Set it as TSS_LIB_PRIVATE_READ_USER / TSS_LIB_PRIVATE_READ_TOKEN on thornode: masked and hidden, protected, and "Expand variable reference" off. Be clear about what that exposes:

   • A non-protected project variable goes to every job of every project pipeline, not only the jobs that source the script. rules: exclusions don't narrow that.
   • Fork MR pipelines that a maintainer runs in this project run the fork's .gitlab-ci.yml with unprotected project variables. Masking stops accidental log output, not deliberate exfiltration. The same applies to branches pushed by make gitlab-trigger-ci.
   • A deploy token reads all refs of tss-lib-private until it expires. It has no per-user audit trail.
   • As a protected variable, it reaches only protected-ref pipelines. MR pipelines for those users then still need step 1.

2. Container registry protection (before merge). Settings → Packages and registries → Protected container repositories: add thorchain/thornode/ci/* with minimum push access Maintainer. Do the same for the thornode release repository. The digest pins above already make a replaced ci/proto-builder tag harmless. This rule also stops a Developer from squatting ci/tools:$VERSION before develop builds it.

3. After merge, check that the develop pipeline's mirror-ci-images and build-ci-tools-image succeeded. Then pin the printed ci/tools digest in the follow-up MR and merge it.

Follow-ups (not in this MR)

• Shallow clones. The project has ci_default_git_depth: 0 and the clone strategy, so every job fetches full history. The failed fetches ran 5–20 minutes before dying. A per-job GIT_DEPTH needs an audit of the jobs that use history (trunk --upstream origin/develop, secret_detection, the fork job).
• Runner pull policy (admin). Allowing ["always", "if-not-present"] lets jobs use a cached image when Docker Hub is down.
• Mocknet chain images pre-built and pulled by digest (brief item 2c). Not started.
• docker:dind --registry-mirror doesn't work with the dependency proxy, whose URLs carry a group path prefix.

Validation

• glab api -X POST projects/13422983/ci/lint with dry_run on develop: valid, no warnings (22 jobs). Also valid merged with !5139 (merged); the only conflict is the default: block (keep both keys).
• trunk check passes on every changed file.
• ci-go-private-auth.sh under bash -e, dash -e and busybox sh -e: both credentials refused (required and warn-only), dedicated token OK, dedicated refused with job token OK.
• A docker build of ci/Dockerfile.regtest up to go mod download fetched tss-lib-private through the go_private_auth secret, and left no .gitconfig behind.
• mirror-ci-images script against a local registry: copies, then skips. The tools image builds locally and has every tool; build/scripts/tests passes in it (38 tests).
• Not run: test-simulation / test-regression end to end. The first pipeline on this MR covers them.

🤖 Generated with Claude Code