fix(df): check and forward plain-HTTP requests through the proxy

Description

Behind the Dependency Firewall, anything fetched over plain HTTP fails. A client sends an http:// URL to its proxy as a plain request rather than CONNECT, and the proxy answered everything but CONNECT with 405, for Maven and every other package manager.

This checks those requests like tunnelled ones, then forwards them.

flowchart LR
    C["package manager"] --> P{"proxy"}
    P -->|CONNECT| T["MITM tunnel"] --> A["admit: match, check, record"]
    P -->|"GET http://host/…"| H["handlePlain"] --> A
    A -->|blocked| B["403"]
    A -->|allowed| U["forward upstream"]
  • Both paths share admit, so blocks, warnings and failed checks behave the same over HTTP and HTTPS, uploads included.
  • Forwarding uses httputil.ReverseProxy on the proxy's own upstream transport, so TLS settings and the outbound proxy still apply: HTTP_PROXY, or HTTPS_PROXY when that's unset, uppercase names first like the rest of glab. Proxy-Authorization and other hop-by-hop headers are dropped.
  • The Upgrade header is removed and an upstream 101 is refused, so the connection can't become an unchecked tunnel. Removing the header rather than refusing the request keeps libcurl clients such as cargo working.
  • A blocked upload gets its 403 before the proxy drains up to 1 MiB of the body. Stop takes at most about 10 s even when a client stalls mid-upload.
  • Like the tunnel, plain HTTP forwards requests no matcher recognises, such as metadata, without a check. That includes glab df go's plain HTTP; Go modules are still checked by glab's own module proxy.
  • The plain-HTTP fallback from !4310 (merged) is removed. A settings file that names an http:// mirror or repository keeps its mirrors, credentials and profiles, and exact matching works for http:// mirrors too.

Decision: plain-HTTP downloads that failed with 405 now succeed after the same check. Protecting them in transit stays with the client, for example lockfile hashes.

golang.org/x/net becomes a direct dependency for httpproxy.Config: the standard library can't send http:// requests through HTTPS_PROXY, and http.ProxyFromEnvironment reads the environment once per process.

!4327 (closed) stacks on this one.

Relates to gitlab-org/gitlab#637331.

How has this been tested?

Real mvn 3.9.12 with an http:// mirrorOf=* mirror and the fake policy checker:

Run !4310 (merged) This MR
-s, no rule falls back to Central with a warning builds through the http:// mirror
-s, rule on org.dom4j:dom4j blocked (from Central) blocked on the plain request
-gs 405 on every download, as released glab builds through the http:// mirror

Also with plain-HTTP registries: mvn deploy, npm install and glab df pip work, and a rule blocks the upload, tarball or wheel with 403.

New tests
  • TestProxyPlainHTTP: blocks, warnings and fail-closed checks over plain HTTP; headers that must not travel are dropped; Upgrade and 101 can't open a tunnel; Stop is bounded; a blocked upload still sees its 403.
  • TestProxyPlainHTTPOutboundProxyFromEnvironment, TestProxyReadsUppercaseProxyVariablesFirst, TestProxyKeepsPlainHTTPDirectInCGI: plain HTTP uses HTTP_PROXY, else HTTPS_PROXY, NO_PROXY skips both, uppercase names win, and CGI stays direct.
  • TestEnvironmentHelp, TestNewCmdEnvironmentHelp: the shared help names HTTP_PROXY; go keeps its own.
  • TestMavenPlainHTTPMirrorIsMerged, TestMavenMirrorKnownURLsOverPlainHTTP: a plain-HTTP mirror stays in the merged settings, and exact matching works for it.
  • TestProxyFailsClosedOnUnclearedMatch, TestProxyRefusesOtherAbsoluteSchemes, TestProxyPlainHTTPClientGoesAway: a match the matcher can't clear blocks on both paths, only http:// is served plain, and a client that leaves cancels its check.

Commands: go test -race ./internal/dependencyfirewall/... ./internal/commands/df/..., go test -count=20 -race -run TestProxyPlainHTTP ./internal/dependencyfirewall/proxy/, golangci-lint run ./internal/dependencyfirewall/..., lefthook run pre-push.

Edited by Hannah Baker

Merge request reports

Loading
Loading