feat(df): fetch Go modules through the user's own GOPROXY

Description

glab df go now fetches allowed modules through the user's own GOPROXY, from the environment or the go env file, instead of proxy.golang.org. It no longer refuses a custom GOPROXY. The child go still talks only to the loopback gate, so every module zip is checked before it's fetched.

Why the fetcher now runs go

goproxy's GoFetcher fetches proxy entries in-process. Measured against the new tests, it:

  • ignores .netrc and GOAUTH (the child got a 500), and handles only a password in the URL
  • fails on scheme-less entries like corp.example, which go accepts
  • asked the next proxy for all three files when only the zip was missing from the first
  • answered an upstream outage with a bare 500 and no reason

The new fetcher runs the user's go for each request (go list -m -json, go list -m -versions -retracted -json, go mod download -json). It runs from an empty directory, with the user's GOPROXY and private-module settings plus the isolation pins (GOMODCACHE, GOFLAGS, GOTOOLCHAIN, GOWORK, GO111MODULE). cmd/go then applies .netrc, GOAUTH, scheme-less and file:// entries, GOINSECURE, and ,/| fallbacks exactly as it would without glab. goproxy still serves, caches and proxies the checksum database.

Errors:

  • A not-found answer stays a 404. This matches the old fetcher, and it keeps go trying shorter module paths for private repos.
  • Any other failure becomes a text/plain 502 carrying go's message, which go prints as "server response".
  • URL userinfo is redacted everywhere it can appear: the 502 body, go's output and the debug log. go itself hides only the password, and tokens are often passed as the user name.

A GOSUMDB with its own URL, or the sum.golang.google.cn alias, is now served through the loopback proxy. Checksum lookups never reach the MITM proxy, which go older than 1.27 on macOS can't trust.

Known limits

  • Credentials for a GOSUMDB URL must be in the URL. .netrc and GOAUTH apply only to GOPROXY, because goproxy fetches the checksum database itself. The help says so.
  • For a GOSUMDB without a URL, goproxy fetches the checksum database directly instead of through the user's GOPROXY first, as go would. An air-gapped network that reaches sum.golang.org only through its proxy fails verification. Follow-up.
  • Each version list or download starts a go process. In the top-10K soak, batches took 1.86 times as long as with !4169 (merged) (median 275s against 127s per 100 modules; 8.4h against 4.5h in total).
  • A GOAUTH command helper's own error output is redacted only for URL userinfo. Anything else the helper prints reaches the 502 body as-is.
  • Not exercised yet: a Nexus Go proxy (testing-plan items 2.4 and 2.5).
  • The plan's U7 steps to normalize GOPROXY entries and handle file:// are dropped, because go does both itself.

How has this been tested?

  • Real-go tests against local upstreams (pm/golang_upstream_test.go, backend tests):
    • a GOPROXY set only with go env -w
    • a,b,direct where b sees only what a returned 404 for
    • | fallthrough
    • a scheme-less entry
    • an upstream that is down (readable error, no fallthrough unless the list says so)
    • a blocked module that the upstream would serve
    • basic auth from .netrc and from user:pass@
    • credentials kept out of verdicts, the CI log, 403 bodies, go's output and the debug log
    • a custom GOSUMDB URL kept on loopback
    • GO111MODULE=off in the environment
    • retracted versions kept in lists
  • Real glab df go on macOS with the fake checker:
Case Result
go env -w GOPROXY=http://127.0.0.1:<recorder> (forwards to proxy.golang.org) downloaded; the recorder saw .info, .mod, .zip
same with GO111MODULE=off exported downloaded
blocked rsc.io/sampler via that recorder refused with the reason
GOPROXY=https://user:s3cr3t@corp.example (unreachable) 502 … Get "https://xxxxx@corp.example/…": no such host; the secret appears nowhere, even with GLAB_DEBUG=true
MR3's earlier smoke cases: allowed, blocked, GOPRIVATE over git, GOPROXY=off, no download unchanged
  • Testing-plan items 1.1–1.6, built from this branch after rebasing onto main. Each run downloads rsc.io/quote@v1.5.2 and rsc.io/sampler@v1.3.0 through local recorders that forward to proxy.golang.org, with the fake checker. HTTPS_PROXY points at a closed port, so anything that bypassed the configured upstream would fail. sum.golang.org was exempt: proxy.golang.org doesn't proxy the checksum database, so go contacts it directly by design.
Item Setup Result
1.1 GOPROXY=http://<recorder> downloaded; both zips came through the recorder
1.2 GOPROXY set only with go env -w downloaded; both zips came through the recorder
1.3 GOPROXY=http://<a>,http://<b>,direct, a answers 404 for zips downloaded; a 404'd both zips and b served them, direct never reached
1.4 HTTPS upstream requiring basic auth, credentials in the URL / in .netrc downloaded both ways with no unauthenticated request; the password appears neither in the output nor in ci-log.json
1.5 GOPROXY=http://127.0.0.1:1 (down) exit 1 with 502 … connection refused; no fall-through
1.6 blocked rsc.io/sampler through the recorder 403 … blocked by the GitLab Dependency Firewall with the reason
  • Toolchain downloads: glab df go version with GOTOOLCHAIN=go1.26.10 (local go is 1.27.3), built from c3de73f26, with the fake checker. Toolchain zips skip the policy check by design, because go verifies them against the checksum database; they still go through the gate and the fetcher.
Case Result
default GOPROXY downloaded go1.26.10 and ran it; no verdict recorded
GOPROXY=http://<recorder>, other egress blocked downloaded; the toolchain zip came through the recorder
policy blocks golang.org/toolchain@v0.0.1-go1.26.10.darwin-arm64 downloaded, as designed
GOTOOLCHAIN=go1.99.0 (doesn't exist) toolchain not available, same as plain go
GOPROXY=off toolchain not available, same as plain go
GOSUMDB=off refused, as plain go refuses: 502 … verifying go.mod: checksum database disabled by GOSUMDB=off

These runs print the nothing-checked notice, which is accurate: no module zip was checked.

  • Top-10K module soak against GDK, compared with !4169 (merged)'s full run. No regression: all 9,988 listed modules end with the same result as on !4169 (merged), apart from one whose advisory data changed between the runs. Batches 1–66 ran on 12a75d98a; batches 67–100, the Ctrl-C runs and the retries ran on c3de73f26.
!4169 (merged) this MR
downloaded 9,795 9,794
blocked by policy 137 138
failed closed 0 0
absent upstream 56 56
missing 0 0
canary blocked 11/11 11/11
Ctrl-C runs passed 5/5 5/5

The harness's own report says FAIL. The causes, all environmental:

  • Two modules recorded as glab bugs, both in batch 57: github.com/reviewdog/reviewdog@v0.21.2 (proxy.golang.org reset the connection) and github.com/cometbft/cometbft@v1.0.1 (read timeout from proxy.golang.org). The Mac slept for 1.6 hours in the middle of that batch. This MR passed the upstream error through as a readable 502, which the harness, written before this MR, counts as a glab bug and never retries. Re-running batch 57 through the same harness, with its fake checker so only the download path is exercised, downloaded all 100 modules on both main (2e352a17e) and this MR (c3de73f26), these two included. The table counts them as downloaded.
  • 3 failed invariant checks: all batch 56, which timed out while the Mac slept. Its 100 modules passed on retry, with the same results as on !4169 (merged).
  • 17 modules failed closed in the first part, during GDK outages right after the wake (Workhorse answered evaluate with 502). They passed on retry, with the same results as on !4169 (merged).

The one real difference is github.com/labstack/echo/v5@v5.3.1, downloaded on !4169 (merged) and blocked here: GDK's advisory data gained a critical vulnerability between the runs, and !4169 (merged)'s build blocked it too when rechecked on 2026-10-26. Every invariant held in batches 67–100.

  • Code review covered correctness, project standards, testing, maintainability, security, reliability and adversarial lenses. Applied: the GO111MODULE pin, retracted versions, redacting goproxy's log lines, and a flat test assertion. Not applied:
    • Narrowing the not-found phrases. It would stop go's module-path probing for private repos; the current mapping is now pinned by tests.
    • Merging two test helpers, because proof tests call them.
Edited by Arpit Gogia

Merge request reports

Loading
Loading