fix(df): harden the proxy's policy checks and shutdown

What does this MR do and why?

Follow-ups from the review of !4326 (merged), split out of !4327 (closed):

  • checkPolicy checks the primary coordinate first, then the alternates in order, up to four at a time, and starts no more once it sees one blocked or failed. It reads the verdicts in candidate order, so the result doesn't depend on which check returns first.
  • A panicking policy check blocks the request instead of ending glab, and says it was glab that failed. A panic leaves no verdict, so glab fails closed as it does when GitLab can't evaluate a package: a glab bug can't let a package skip the policy check. The CI log entry and the end-of-job summary say "glab hit an internal error while checking @ and blocked it…", and the 403 says "Blocked: glab internal error while checking @". The warning names the coordinate glab was checking and leaves out "and blocked it", because on a mirrored path that coordinate can be an alternate that didn't decide the verdict. CachingChecker, checkPolicy and the Go module proxy all recover, and a retry runs the check again.
  • Stop drains plain-HTTP handlers as well as MITM tunnels, so a Block recorded after Shutdown gives up on a stalled upload still reaches the CI log (Rane's thread).
  • The help text says the proxy isn't an egress filter (Rane's thread).

What it looks like

glab-internal-error-demo

glab df maven in this demo job, built from this MR with the policy check for log4j-core patched to panic. The panic is injected for the demo; it isn't something this MR introduces. The job ran an earlier head, so its warning line still ends in "and blocked it".

How has this been tested?

Tests
  • TestCachePanicFailsClosed: the caller that ran a panicking check doesn't see the panic; it and the cache entry a waiter reads are Blocked with CheckFailed, the warning names the checked package, and the next call runs the check again.
  • TestPanickingCheckBlocksInsteadOfCrashing (Go module proxy): with a panicking checker, raw or behind the cache, the module gets a 403, never reaches the backend, and the panic value stays out of the response.
  • TestPanickingCheckBlockResponseNamesTheInternalError, TestInternalErrorStatusLineCannotInjectHeaders, TestInternalErrorTextIsSanitized, TestPanickingAlternateNamesTheRequest: the internal-error text, sanitized and capped: the 403 names the requested package, and the warning names the coordinate it checked.
  • TestProxyStopDrainsPlainHandlerBeforeVerdicts: a plain-HTTP handler that records after Shutdown times out still lands in Verdicts().
  • TestMavenMirrorPolicyMatching (moved to mavenmatch_prefixes_test.go): the concurrency cap, a blocked or failing primary, no new checks after a failing alternate, a panicking check, and verdict order.
  • Each fix was removed in turn, and the matching test failed.

go test -race ./internal/dependencyfirewall/... ./internal/commands/df/..., golangci-lint run, make gen-docs (no diff).

Relates to https://gitlab.com/gitlab-org/gitlab/-/work_items/637331.

Edited by Hannah Baker

Merge request reports

Loading
Loading