fix(df): check and forward plain-HTTP requests through the proxy
Description
Behind the Dependency Firewall, anything fetched over plain HTTP fails. A client sends an http:// URL to its proxy as a plain request rather than CONNECT, and the proxy answered everything but CONNECT with 405, for Maven and every other package manager.
This checks those requests like tunnelled ones, then forwards them.
flowchart LR
C["package manager"] --> P{"proxy"}
P -->|CONNECT| T["MITM tunnel"] --> A["admit: match, check, record"]
P -->|"GET http://host/…"| H["handlePlain"] --> A
A -->|blocked| B["403"]
A -->|allowed| U["forward upstream"]- Both paths share
admit, so blocks, warnings and failed checks behave the same over HTTP and HTTPS, uploads included. - Forwarding uses
httputil.ReverseProxyon the proxy's own upstream transport, so TLS settings and the outbound proxy still apply:HTTP_PROXY, orHTTPS_PROXYwhen that's unset, uppercase names first like the rest of glab.Proxy-Authorizationand other hop-by-hop headers are dropped. - The
Upgradeheader is removed and an upstream101is refused, so the connection can't become an unchecked tunnel. Removing the header rather than refusing the request keeps libcurl clients such as cargo working. - A blocked upload gets its
403before the proxy drains up to 1 MiB of the body.Stoptakes at most about 10 s even when a client stalls mid-upload. - Like the tunnel, plain HTTP forwards requests no matcher recognises, such as metadata, without a check. That includes
glab df go's plain HTTP; Go modules are still checked by glab's own module proxy. - The plain-HTTP fallback from !4310 (merged) is removed. A settings file that names an
http://mirror or repository keeps its mirrors, credentials and profiles, and exact matching works forhttp://mirrors too.
Decision: plain-HTTP downloads that failed with 405 now succeed after the same check. Protecting them in transit stays with the client, for example lockfile hashes.
golang.org/x/net becomes a direct dependency for httpproxy.Config: the standard library can't send http:// requests through HTTPS_PROXY, and http.ProxyFromEnvironment reads the environment once per process.
!4327 (closed) stacks on this one.
Related Issues
Relates to gitlab-org/gitlab#637331.
How has this been tested?
Real mvn 3.9.12 with an http:// mirrorOf=* mirror and the fake policy checker:
| Run | !4310 (merged) | This MR |
|---|---|---|
-s, no rule |
falls back to Central with a warning | builds through the http:// mirror |
-s, rule on org.dom4j:dom4j |
blocked (from Central) | blocked on the plain request |
-gs |
405 on every download, as released glab |
builds through the http:// mirror |
Also with plain-HTTP registries: mvn deploy, npm install and glab df pip work, and a rule blocks the upload, tarball or wheel with 403.
New tests
TestProxyPlainHTTP: blocks, warnings and fail-closed checks over plain HTTP; headers that must not travel are dropped;Upgradeand101can't open a tunnel;Stopis bounded; a blocked upload still sees its403.TestProxyPlainHTTPOutboundProxyFromEnvironment,TestProxyReadsUppercaseProxyVariablesFirst,TestProxyKeepsPlainHTTPDirectInCGI: plain HTTP usesHTTP_PROXY, elseHTTPS_PROXY,NO_PROXYskips both, uppercase names win, and CGI stays direct.TestEnvironmentHelp,TestNewCmdEnvironmentHelp: the shared help namesHTTP_PROXY;gokeeps its own.TestMavenPlainHTTPMirrorIsMerged,TestMavenMirrorKnownURLsOverPlainHTTP: a plain-HTTP mirror stays in the merged settings, and exact matching works for it.TestProxyFailsClosedOnUnclearedMatch,TestProxyRefusesOtherAbsoluteSchemes,TestProxyPlainHTTPClientGoesAway: a match the matcher can't clear blocks on both paths, onlyhttp://is served plain, and a client that leaves cancels its check.
Commands: go test -race ./internal/dependencyfirewall/... ./internal/commands/df/..., go test -count=20 -race -run TestProxyPlainHTTP ./internal/dependencyfirewall/proxy/, golangci-lint run ./internal/dependencyfirewall/..., lefthook run pre-push.