feat(df): fetch Go modules through the user's own GOPROXY
Description
glab df go now fetches allowed modules through the user's own GOPROXY, from the environment or the go env file, instead of proxy.golang.org. It no longer refuses a custom GOPROXY. The child go still talks only to the loopback gate, so every module zip is checked before it's fetched.
Why the fetcher now runs go
goproxy's GoFetcher fetches proxy entries in-process. Measured against the new tests, it:
- ignores
.netrcandGOAUTH(the child got a 500), and handles only a password in the URL - fails on scheme-less entries like
corp.example, whichgoaccepts - asked the next proxy for all three files when only the zip was missing from the first
- answered an upstream outage with a bare
500and no reason
The new fetcher runs the user's go for each request (go list -m -json, go list -m -versions -retracted -json, go mod download -json). It runs from an empty directory, with the user's GOPROXY and private-module settings plus the isolation pins (GOMODCACHE, GOFLAGS, GOTOOLCHAIN, GOWORK, GO111MODULE). cmd/go then applies .netrc, GOAUTH, scheme-less and file:// entries, GOINSECURE, and ,/| fallbacks exactly as it would without glab. goproxy still serves, caches and proxies the checksum database.
Errors:
- A not-found answer stays a 404. This matches the old fetcher, and it keeps
gotrying shorter module paths for private repos. - Any other failure becomes a
text/plain502 carryinggo's message, whichgoprints as "server response". - URL userinfo is redacted everywhere it can appear: the 502 body,
go's output and the debug log.goitself hides only the password, and tokens are often passed as the user name.
A GOSUMDB with its own URL, or the sum.golang.google.cn alias, is now served through the loopback proxy. Checksum lookups never reach the MITM proxy, which go older than 1.27 on macOS can't trust.
Known limits
- Credentials for a
GOSUMDBURL must be in the URL..netrcandGOAUTHapply only toGOPROXY, because goproxy fetches the checksum database itself. The help says so. - For a
GOSUMDBwithout a URL, goproxy fetches the checksum database directly instead of through the user'sGOPROXYfirst, asgowould. An air-gapped network that reachessum.golang.orgonly through its proxy fails verification. Follow-up. - Each version list or download starts a
goprocess. In the top-10K soak, batches took 1.86 times as long as with !4169 (merged) (median 275s against 127s per 100 modules; 8.4h against 4.5h in total). - A
GOAUTHcommand helper's own error output is redacted only for URL userinfo. Anything else the helper prints reaches the 502 body as-is. - Not exercised yet: a Nexus Go proxy (testing-plan items 2.4 and 2.5).
- The plan's U7 steps to normalize
GOPROXYentries and handlefile://are dropped, becausegodoes both itself.
How has this been tested?
- Real-
gotests against local upstreams (pm/golang_upstream_test.go, backend tests):- a
GOPROXYset only withgo env -w a,b,directwherebsees only whatareturned 404 for|fallthrough- a scheme-less entry
- an upstream that is down (readable error, no fallthrough unless the list says so)
- a blocked module that the upstream would serve
- basic auth from
.netrcand fromuser:pass@ - credentials kept out of verdicts, the CI log, 403 bodies,
go's output and the debug log - a custom
GOSUMDBURL kept on loopback GO111MODULE=offin the environment- retracted versions kept in lists
- a
- Real
glab df goon macOS with the fake checker:
| Case | Result |
|---|---|
go env -w GOPROXY=http://127.0.0.1:<recorder> (forwards to proxy.golang.org) |
downloaded; the recorder saw .info, .mod, .zip |
same with GO111MODULE=off exported |
downloaded |
blocked rsc.io/sampler via that recorder |
refused with the reason |
GOPROXY=https://user:s3cr3t@corp.example (unreachable) |
502 … Get "https://xxxxx@corp.example/…": no such host; the secret appears nowhere, even with GLAB_DEBUG=true |
MR3's earlier smoke cases: allowed, blocked, GOPRIVATE over git, GOPROXY=off, no download |
unchanged |
- Testing-plan items 1.1–1.6, built from this branch after rebasing onto
main. Each run downloadsrsc.io/quote@v1.5.2andrsc.io/sampler@v1.3.0through local recorders that forward to proxy.golang.org, with the fake checker.HTTPS_PROXYpoints at a closed port, so anything that bypassed the configured upstream would fail. sum.golang.org was exempt: proxy.golang.org doesn't proxy the checksum database, sogocontacts it directly by design.
| Item | Setup | Result |
|---|---|---|
| 1.1 | GOPROXY=http://<recorder> |
downloaded; both zips came through the recorder |
| 1.2 | GOPROXY set only with go env -w |
downloaded; both zips came through the recorder |
| 1.3 | GOPROXY=http://<a>,http://<b>,direct, a answers 404 for zips |
downloaded; a 404'd both zips and b served them, direct never reached |
| 1.4 | HTTPS upstream requiring basic auth, credentials in the URL / in .netrc |
downloaded both ways with no unauthenticated request; the password appears neither in the output nor in ci-log.json |
| 1.5 | GOPROXY=http://127.0.0.1:1 (down) |
exit 1 with 502 … connection refused; no fall-through |
| 1.6 | blocked rsc.io/sampler through the recorder |
403 … blocked by the GitLab Dependency Firewall with the reason |
- Toolchain downloads:
glab df go versionwithGOTOOLCHAIN=go1.26.10(localgois 1.27.3), built fromc3de73f26, with the fake checker. Toolchain zips skip the policy check by design, becausegoverifies them against the checksum database; they still go through the gate and the fetcher.
| Case | Result |
|---|---|
default GOPROXY |
downloaded go1.26.10 and ran it; no verdict recorded |
GOPROXY=http://<recorder>, other egress blocked |
downloaded; the toolchain zip came through the recorder |
policy blocks golang.org/toolchain@v0.0.1-go1.26.10.darwin-arm64 |
downloaded, as designed |
GOTOOLCHAIN=go1.99.0 (doesn't exist) |
toolchain not available, same as plain go |
GOPROXY=off |
toolchain not available, same as plain go |
GOSUMDB=off |
refused, as plain go refuses: 502 … verifying go.mod: checksum database disabled by GOSUMDB=off |
These runs print the nothing-checked notice, which is accurate: no module zip was checked.
- Top-10K module soak against GDK, compared with !4169 (merged)'s full run. No regression: all 9,988 listed modules end with the same result as on !4169 (merged), apart from one whose advisory data changed between the runs. Batches 1–66 ran on
12a75d98a; batches 67–100, the Ctrl-C runs and the retries ran onc3de73f26.
| !4169 (merged) | this MR | |
|---|---|---|
| downloaded | 9,795 | 9,794 |
| blocked by policy | 137 | 138 |
| failed closed | 0 | 0 |
| absent upstream | 56 | 56 |
| missing | 0 | 0 |
| canary blocked | 11/11 | 11/11 |
| Ctrl-C runs passed | 5/5 | 5/5 |
The harness's own report says FAIL. The causes, all environmental:
- Two modules recorded as glab bugs, both in batch 57:
github.com/reviewdog/reviewdog@v0.21.2(proxy.golang.org reset the connection) andgithub.com/cometbft/cometbft@v1.0.1(read timeout from proxy.golang.org). The Mac slept for 1.6 hours in the middle of that batch. This MR passed the upstream error through as a readable 502, which the harness, written before this MR, counts as a glab bug and never retries. Re-running batch 57 through the same harness, with its fake checker so only the download path is exercised, downloaded all 100 modules on bothmain(2e352a17e) and this MR (c3de73f26), these two included. The table counts them as downloaded. - 3 failed invariant checks: all batch 56, which timed out while the Mac slept. Its 100 modules passed on retry, with the same results as on !4169 (merged).
- 17 modules failed closed in the first part, during GDK outages right after the wake (Workhorse answered
evaluatewith 502). They passed on retry, with the same results as on !4169 (merged).
The one real difference is github.com/labstack/echo/v5@v5.3.1, downloaded on !4169 (merged) and blocked here: GDK's advisory data gained a critical vulnerability between the runs, and !4169 (merged)'s build blocked it too when rechecked on 2026-10-26. Every invariant held in batches 67–100.
- Code review covered correctness, project standards, testing, maintainability, security, reliability and adversarial lenses. Applied: the
GO111MODULEpin, retracted versions, redacting goproxy's log lines, and a flat test assertion. Not applied:- Narrowing the not-found phrases. It would stop
go's module-path probing for private repos; the current mapping is now pinned by tests. - Merging two test helpers, because proof tests call them.
- Narrowing the not-found phrases. It would stop