feat(df): reach registries through the package manager's own proxy
Description
This MR is stacked on !4326 (merged), which merges first. It targets that branch until then, so the diff shows only the changes in this MR.
glab df points each package manager's own proxy variable at the inspection proxy, so the package manager can't route around it. A corporate proxy the user had set in one of those variables was then lost. !4150 (merged) (merged) makes the inspection proxy use HTTPS_PROXY for its registry connection, but a user whose proxy is only in, for example, PIP_PROXY still got a failed install.
This MR reads the package manager's own proxy from the environment before overriding it, and uses it for the inspection proxy's upstream connection. It follows the same rules the package manager does, which I measured against each tool by pointing every variable at a different local listener:
| Package manager | Proxy used for HTTPS, highest first | Hosts reached directly |
|---|---|---|
| npm 10.9 and 11.17, pnpm 10.12 | npm_config_https_proxy, npm_config_proxy, HTTPS_PROXY (keys match case-insensitively; a value of false, null or undefined counts as unset) |
npm_config_noproxy and NO_PROXY, combined |
| Yarn 4.9 (Berry) | YARN_HTTPS_PROXY only |
None: NO_PROXY is ignored |
| Yarn 1.22 (Classic) | YARN_HTTPS_PROXY, npm_config_https_proxy, npm_config_proxy, HTTPS_PROXY |
None: NO_PROXY is ignored once a Yarn or npm proxy is set |
| pip 24.3, pipenv | PIP_PROXY, then HTTPS_PROXY |
PIP_PROXY ignores NO_PROXY |
| cargo 1.99 | CARGO_HTTP_PROXY (scheme optional), then HTTPS_PROXY |
NO_PROXY or no_proxy |
The other package managers (poetry, uv, twine, gem, bundle, maven, gradle) read no proxy variable of their own, so they keep the default from !4326 (merged): HTTPS_PROXY, and HTTP_PROXY for plain-HTTP registries.
Changes:
- A package manager can implement the unexported
upstreamProxyResolverto report its own proxy.Runresolves it from the original environment and passes it to the inspection proxy withproxy.WithUpstreamProxyFor, one option per URL scheme, next to the upstream-failure warning option from !4278 (merged). A scheme without a manager setting keeps the default. - Plain-HTTP registries, which !4326 (merged) forwards, use the same proxy as HTTPS for npm, pnpm, pip and cargo, as those tools do. Yarn Classic uses
npm_config_proxy, thenHTTP_PROXY, for plain HTTP, so Yarn has a separate plain-HTTP resolver. Yarn readsYARN_HTTPS_PROXYin any case, and so does the resolver. - The help for npm, pnpm, Yarn, pip, pipenv and cargo names their proxy variables, and the
glab dependency-firewallhelp says glab reads them first. - No-proxy matching uses
golang.org/x/net/http/httpproxy, the same codehttp.ProxyFromEnvironmentuses.golang.org/x/netwas already an indirect dependency and is now direct. - An invalid proxy URL or unsupported scheme fails the run before the package manager starts. The error names the variable but not its value, because a proxy URL can contain credentials. For npm and pnpm, that now includes a malformed
HTTPS_PROXY, which used to fail each registry request instead. - npm reads
npm_config_*values the way it reads.npmrc, sonpm_config_proxy=false(ornull,undefined) turns the setting off and npm moves on to the next one. The npm and Yarn Classic resolvers do the same, instead of dialling a proxy host namedfalse. - Yarn has no reliable version check, so a Berry user with only
npm_config_https_proxyset now gets that proxy for the registry connection, although Berry itself would ignore it. That is a proxy the user configured, so I treated it as acceptable.
Out of scope, next in the stack:
- Proxies set only in config files (
.npmrc,pip.conf,.yarnrc.yml,.cargo/config.toml, gemrc, Mavensettings.xml,gradle.properties).
This MR reads environment variables only, so it is partial progress on B4 in gitlab-org&24327. B4 stays open until the config-file MR lands.
Related Issues
Related to #8635 (closed) Related to gitlab-org&24327
How has this been tested?
upstream_test.go:- Resolvers: a table test for each resolver's precedence, case folding, empty-as-unset,
false/null-as-unset and no-proxy rules, including Yarn Classic preferringnpm_config_https_proxy, cargo preferringNO_PROXYoverno_proxy, and a case-foldedNPM_CONFIG_NOPROXY. The managers with no proxy setting of their own, Go included, have no resolver. - Proxy choice: a request to a no-proxy host goes direct, and a scheme-less proxy URL works.
- Proxy URLs:
https,socks5andsocks5hproxies are accepted. A malformed URL, an empty host and anftpscheme each fail with their own message, which never contains the URL, so credentials don't leak. An invalidPIP_PROXYstopsRunbefore the package manager starts. - End to end:
Runwithnpm_config_https_proxypointing at a recording proxy sends the registryCONNECTto that proxy. Without the change inRun, this test fails because the request goes direct. - Plain HTTP:
Runsends a plain-HTTP request through Yarn'snpm_config_proxybut notYARN_HTTPS_PROXYorYARN_HTTP_PROXY, falls back toHTTP_PROXYfor Yarn, and uses npm'snpm_config_https_proxyfor plain HTTP too.TestWithUpstreamProxyForOverridesOnlyItsSchemecovers the per-scheme option, andTestOwnProxyVariablesAreReadchecks that every variable named in the help is read.
- Resolvers: a table test for each resolver's precedence, case folding, empty-as-unset,
- Against npm 11.17.0 with a local listener as
HTTPS_PROXY:npm_config_proxy=false,npm_config_https_proxy=falseandnpm_config_https_proxy=nulleach still sendCONNECTtoHTTPS_PROXY. WithNO_PROXYandnpm_config_noproxynaming different hosts, a host in either list is reached directly. The threefalse/nulltable rows fail if the resolvers take those values literally. - With real package managers and one closed-port proxy per variable, glab dials the same proxy as the bare tool for plain HTTP and HTTPS: Yarn 1.22.19, Yarn 4.9.2 (HTTPS), npm 10.9.8, pnpm 10.12.4, pip 26.0 and cargo 1.98.
go test -race ./internal/dependencyfirewall/... ./internal/commands/df/...passes after mergingmain, and again on 0e3e69bb with./cmd/gen-docs/...added, wheremake gen-docsleaves no diff.golangci-lintreports 0 issues, andscripts/comment-overlapreports 0 findings for the changed files.
Manual testing
These steps use bash/zsh syntax; in fish, run zsh first. They use mitmproxy in Docker as the corporate proxy and the fake policy checker (GLAB_DF_FAKE_*), so you don't need a project with the dependency firewall enabled. The corporate proxy is set only in each package manager's own variable, never in HTTPS_PROXY, except in step 1c. seen <host> counts the requests the corporate proxy has handled for a host, so a count that goes up means glab sent the registry traffic through it.
Setup. You need Docker, Go, Node.js with npm, and Python 3 with venv (on Debian and Ubuntu, the python3-venv package), plus internet access to the npm registry and PyPI and a free port 8081. Build glab from the root of this branch, then create an empty project to install into. Run every step in that new directory, not in an existing project.
make build
export GLAB=$PWD/bin/glab
mkdir -p ~/df-proxy-test-4156/mitm && cd ~/df-proxy-test-4156
npm init -y >/dev/null
python3 -m venv .venv && export PATH=$PWD/.venv/bin:$PATH
# glab df takes the project from the git remote. The fake checker never contacts it, so a placeholder works.
git init -q && git remote add origin https://gitlab.com/example/df-proxy-test.git
git config remote.origin.glab-resolved-base base
export GLAB_SEND_TELEMETRY=false GLAB_DF_FAKE_DEFAULT=allow PIP_DISABLE_PIP_VERSION_CHECK=1
unset HTTPS_PROXY https_proxy HTTP_PROXY http_proxy NO_PROXY no_proxy
# npm serves cached tarballs without asking the proxy, so use a fresh cache for every install.
fresh() { rm -rf node_modules package-lock.json; export npm_config_cache=$(mktemp -d); }
docker run -d --name corp-proxy-4156 -p 8081:8080 -v ~/df-proxy-test-4156/mitm:/home/mitmproxy/.mitmproxy \
mitmproxy/mitmproxy mitmdump --set block_global=false
CA=~/df-proxy-test-4156/mitm/mitmproxy-ca-cert.pem
seen() { docker logs corp-proxy-4156 2>&1 | grep -c " GET https://$1/"; }1. npm
| Step | Command | Expected |
|---|---|---|
| 1a | fresh; npm_config_https_proxy=http://127.0.0.1:8081 $GLAB df npm install left-pad |
The install fails after about a minute, because npm retries, and glab prints warning: dependency firewall request to registry registry.npmjs.org:443 failed: tls: failed to verify certificate: x509: .... The registry connection went to the corporate proxy, which glab doesn't trust yet. On main, the install succeeds, because registry traffic skips npm's proxy. |
| 1b | fresh; GLAB_CA_CERT=$CA npm_config_https_proxy=http://127.0.0.1:8081 $GLAB df npm install left-pad; seen registry.npmjs.org |
Installs, and the count goes up by 2: the package metadata and the tarball. On main, it installs and the count doesn't go up. |
| 1c | fresh; GLAB_CA_CERT=$CA HTTPS_PROXY=http://127.0.0.1:8081 npm_config_proxy=false $GLAB df npm install left-pad; seen registry.npmjs.org |
Installs, and the count goes up: npm_config_proxy=false means "no npm proxy", so glab falls back to HTTPS_PROXY, as npm does. |
| 1d | fresh; npm_config_https_proxy=http://127.0.0.1:8081 npm_config_noproxy=registry.npmjs.org $GLAB df npm install left-pad; seen registry.npmjs.org |
Installs with no CA set, and the count doesn't change: hosts in npm's own noproxy list are reached directly. |
2. pip
| Step | Command | Expected |
|---|---|---|
| 2a | GLAB_CA_CERT=$CA PIP_PROXY=http://127.0.0.1:8081 $GLAB df pip install --force-reinstall --no-cache-dir six; seen files.pythonhosted.org |
Installs, and the count goes up by 2. On main, it installs and the count doesn't go up. |
| 2b | PIP_PROXY=ftp://corp:s3cret@127.0.0.1:2121 $GLAB df pip install six |
Fails within a few seconds with invalid outbound proxy in PIP_PROXY: unsupported proxy scheme "ftp", before pip starts. The password isn't printed. On main, it installs. |
Clean up with docker rm -f corp-proxy-4156; rm -rf ~/df-proxy-test-4156; unset GLAB_DF_FAKE_DEFAULT, then open a new terminal to reset PATH.