feat(df): reach registries through the package manager's own proxy

Description

This MR is stacked on !4326 (merged), which merges first. It targets that branch until then, so the diff shows only the changes in this MR.

glab df points each package manager's own proxy variable at the inspection proxy, so the package manager can't route around it. A corporate proxy the user had set in one of those variables was then lost. !4150 (merged) (merged) makes the inspection proxy use HTTPS_PROXY for its registry connection, but a user whose proxy is only in, for example, PIP_PROXY still got a failed install.

This MR reads the package manager's own proxy from the environment before overriding it, and uses it for the inspection proxy's upstream connection. It follows the same rules the package manager does, which I measured against each tool by pointing every variable at a different local listener:

Package manager Proxy used for HTTPS, highest first Hosts reached directly
npm 10.9 and 11.17, pnpm 10.12 npm_config_https_proxy, npm_config_proxy, HTTPS_PROXY (keys match case-insensitively; a value of false, null or undefined counts as unset) npm_config_noproxy and NO_PROXY, combined
Yarn 4.9 (Berry) YARN_HTTPS_PROXY only None: NO_PROXY is ignored
Yarn 1.22 (Classic) YARN_HTTPS_PROXY, npm_config_https_proxy, npm_config_proxy, HTTPS_PROXY None: NO_PROXY is ignored once a Yarn or npm proxy is set
pip 24.3, pipenv PIP_PROXY, then HTTPS_PROXY PIP_PROXY ignores NO_PROXY
cargo 1.99 CARGO_HTTP_PROXY (scheme optional), then HTTPS_PROXY NO_PROXY or no_proxy

The other package managers (poetry, uv, twine, gem, bundle, maven, gradle) read no proxy variable of their own, so they keep the default from !4326 (merged): HTTPS_PROXY, and HTTP_PROXY for plain-HTTP registries.

Changes:

  • A package manager can implement the unexported upstreamProxyResolver to report its own proxy. Run resolves it from the original environment and passes it to the inspection proxy with proxy.WithUpstreamProxyFor, one option per URL scheme, next to the upstream-failure warning option from !4278 (merged). A scheme without a manager setting keeps the default.
  • Plain-HTTP registries, which !4326 (merged) forwards, use the same proxy as HTTPS for npm, pnpm, pip and cargo, as those tools do. Yarn Classic uses npm_config_proxy, then HTTP_PROXY, for plain HTTP, so Yarn has a separate plain-HTTP resolver. Yarn reads YARN_HTTPS_PROXY in any case, and so does the resolver.
  • The help for npm, pnpm, Yarn, pip, pipenv and cargo names their proxy variables, and the glab dependency-firewall help says glab reads them first.
  • No-proxy matching uses golang.org/x/net/http/httpproxy, the same code http.ProxyFromEnvironment uses. golang.org/x/net was already an indirect dependency and is now direct.
  • An invalid proxy URL or unsupported scheme fails the run before the package manager starts. The error names the variable but not its value, because a proxy URL can contain credentials. For npm and pnpm, that now includes a malformed HTTPS_PROXY, which used to fail each registry request instead.
  • npm reads npm_config_* values the way it reads .npmrc, so npm_config_proxy=false (or null, undefined) turns the setting off and npm moves on to the next one. The npm and Yarn Classic resolvers do the same, instead of dialling a proxy host named false.
  • Yarn has no reliable version check, so a Berry user with only npm_config_https_proxy set now gets that proxy for the registry connection, although Berry itself would ignore it. That is a proxy the user configured, so I treated it as acceptable.

Out of scope, next in the stack:

  • Proxies set only in config files (.npmrc, pip.conf, .yarnrc.yml, .cargo/config.toml, gemrc, Maven settings.xml, gradle.properties).

This MR reads environment variables only, so it is partial progress on B4 in gitlab-org&24327. B4 stays open until the config-file MR lands.

Related to #8635 (closed) Related to gitlab-org&24327

How has this been tested?

  • upstream_test.go:
    • Resolvers: a table test for each resolver's precedence, case folding, empty-as-unset, false/null-as-unset and no-proxy rules, including Yarn Classic preferring npm_config_https_proxy, cargo preferring NO_PROXY over no_proxy, and a case-folded NPM_CONFIG_NOPROXY. The managers with no proxy setting of their own, Go included, have no resolver.
    • Proxy choice: a request to a no-proxy host goes direct, and a scheme-less proxy URL works.
    • Proxy URLs: https, socks5 and socks5h proxies are accepted. A malformed URL, an empty host and an ftp scheme each fail with their own message, which never contains the URL, so credentials don't leak. An invalid PIP_PROXY stops Run before the package manager starts.
    • End to end: Run with npm_config_https_proxy pointing at a recording proxy sends the registry CONNECT to that proxy. Without the change in Run, this test fails because the request goes direct.
    • Plain HTTP: Run sends a plain-HTTP request through Yarn's npm_config_proxy but not YARN_HTTPS_PROXY or YARN_HTTP_PROXY, falls back to HTTP_PROXY for Yarn, and uses npm's npm_config_https_proxy for plain HTTP too. TestWithUpstreamProxyForOverridesOnlyItsScheme covers the per-scheme option, and TestOwnProxyVariablesAreRead checks that every variable named in the help is read.
  • Against npm 11.17.0 with a local listener as HTTPS_PROXY: npm_config_proxy=false, npm_config_https_proxy=false and npm_config_https_proxy=null each still send CONNECT to HTTPS_PROXY. With NO_PROXY and npm_config_noproxy naming different hosts, a host in either list is reached directly. The three false/null table rows fail if the resolvers take those values literally.
  • With real package managers and one closed-port proxy per variable, glab dials the same proxy as the bare tool for plain HTTP and HTTPS: Yarn 1.22.19, Yarn 4.9.2 (HTTPS), npm 10.9.8, pnpm 10.12.4, pip 26.0 and cargo 1.98.
  • go test -race ./internal/dependencyfirewall/... ./internal/commands/df/... passes after merging main, and again on 0e3e69bb with ./cmd/gen-docs/... added, where make gen-docs leaves no diff. golangci-lint reports 0 issues, and scripts/comment-overlap reports 0 findings for the changed files.

Manual testing

These steps use bash/zsh syntax; in fish, run zsh first. They use mitmproxy in Docker as the corporate proxy and the fake policy checker (GLAB_DF_FAKE_*), so you don't need a project with the dependency firewall enabled. The corporate proxy is set only in each package manager's own variable, never in HTTPS_PROXY, except in step 1c. seen <host> counts the requests the corporate proxy has handled for a host, so a count that goes up means glab sent the registry traffic through it.

Setup. You need Docker, Go, Node.js with npm, and Python 3 with venv (on Debian and Ubuntu, the python3-venv package), plus internet access to the npm registry and PyPI and a free port 8081. Build glab from the root of this branch, then create an empty project to install into. Run every step in that new directory, not in an existing project.

make build
export GLAB=$PWD/bin/glab
mkdir -p ~/df-proxy-test-4156/mitm && cd ~/df-proxy-test-4156
npm init -y >/dev/null
python3 -m venv .venv && export PATH=$PWD/.venv/bin:$PATH
# glab df takes the project from the git remote. The fake checker never contacts it, so a placeholder works.
git init -q && git remote add origin https://gitlab.com/example/df-proxy-test.git
git config remote.origin.glab-resolved-base base
export GLAB_SEND_TELEMETRY=false GLAB_DF_FAKE_DEFAULT=allow PIP_DISABLE_PIP_VERSION_CHECK=1
unset HTTPS_PROXY https_proxy HTTP_PROXY http_proxy NO_PROXY no_proxy
# npm serves cached tarballs without asking the proxy, so use a fresh cache for every install.
fresh() { rm -rf node_modules package-lock.json; export npm_config_cache=$(mktemp -d); }
docker run -d --name corp-proxy-4156 -p 8081:8080 -v ~/df-proxy-test-4156/mitm:/home/mitmproxy/.mitmproxy \
  mitmproxy/mitmproxy mitmdump --set block_global=false
CA=~/df-proxy-test-4156/mitm/mitmproxy-ca-cert.pem
seen() { docker logs corp-proxy-4156 2>&1 | grep -c " GET https://$1/"; }

1. npm

Step Command Expected
1a fresh; npm_config_https_proxy=http://127.0.0.1:8081 $GLAB df npm install left-pad The install fails after about a minute, because npm retries, and glab prints warning: dependency firewall request to registry registry.npmjs.org:443 failed: tls: failed to verify certificate: x509: .... The registry connection went to the corporate proxy, which glab doesn't trust yet. On main, the install succeeds, because registry traffic skips npm's proxy.
1b fresh; GLAB_CA_CERT=$CA npm_config_https_proxy=http://127.0.0.1:8081 $GLAB df npm install left-pad; seen registry.npmjs.org Installs, and the count goes up by 2: the package metadata and the tarball. On main, it installs and the count doesn't go up.
1c fresh; GLAB_CA_CERT=$CA HTTPS_PROXY=http://127.0.0.1:8081 npm_config_proxy=false $GLAB df npm install left-pad; seen registry.npmjs.org Installs, and the count goes up: npm_config_proxy=false means "no npm proxy", so glab falls back to HTTPS_PROXY, as npm does.
1d fresh; npm_config_https_proxy=http://127.0.0.1:8081 npm_config_noproxy=registry.npmjs.org $GLAB df npm install left-pad; seen registry.npmjs.org Installs with no CA set, and the count doesn't change: hosts in npm's own noproxy list are reached directly.

2. pip

Step Command Expected
2a GLAB_CA_CERT=$CA PIP_PROXY=http://127.0.0.1:8081 $GLAB df pip install --force-reinstall --no-cache-dir six; seen files.pythonhosted.org Installs, and the count goes up by 2. On main, it installs and the count doesn't go up.
2b PIP_PROXY=ftp://corp:s3cret@127.0.0.1:2121 $GLAB df pip install six Fails within a few seconds with invalid outbound proxy in PIP_PROXY: unsupported proxy scheme "ftp", before pip starts. The password isn't printed. On main, it installs.

Clean up with docker rm -f corp-proxy-4156; rm -rf ~/df-proxy-test-4156; unset GLAB_DF_FAKE_DEFAULT, then open a new terminal to reset PATH.

Edited by Hannah Baker

Merge request reports

Loading
Loading