perf(client): tune HTTP transport for concurrent same-host collection

Problem

The GitLab REST client built a bare &http.Client{Timeout: ...}, so it inherited http.DefaultTransport, whose MaxIdleConnsPerHost defaults to 2. The collector fans out concurrent requests (the --concurrency worker count) against a single host: the GitLab instance. With only 2 idle connections kept per host, every concurrent wave beyond 2 closes and reopens a Transmission Control Protocol (TCP) plus Transport Layer Security (TLS) connection on the next request: re-dial, re-handshake, every time. At 1000-user scale that is a real throughput and latency drag.

Fix

Install a tuned *http.Transport cloned from http.DefaultTransport via Clone(), which preserves proxy-from-environment, dial timeouts, ForceAttemptHTTP2, and the TLS defaults (a hand-built Transport would lose these), then raise the idle-connection caps:

  • MaxIdleConnsPerHost = max(concurrency, 8). Derived from the collector worker count so the idle pool scales when concurrency is raised, with a floor of 8 so a default (concurrency 1) run still keeps a useful pool. collect and run pass the configured worker count through a new WithConcurrency option.
  • MaxIdleConns (total) = 100. manifold talks to one host, so this only needs headroom above the per-host cap.
  • MaxConnsPerHost left unset (0, unlimited) so it never throttles below the worker count.
  • IdleConnTimeout keeps the 90s clone default, long enough to span the gap between collection waves.

WithHTTPClient (used by tests) bypasses the tuned transport: a caller-supplied client owns its transport.

Body drain

Idle-connection reuse requires every response body to be fully read and closed. GetJSON, PostJSON, PutJSON already drain and close. Delete did not: it returned the raw response, and both seeder callers ignored it and never closed the body, pinning the connection and defeating reuse on that path. Delete now drains and closes the body before returning.

Test proof

  • TestNew_TunedTransportFloor / TestNew_TunedTransportScalesWithConcurrency: assert Transport.(*http.Transport).MaxIdleConnsPerHost equals the tuned value and is >= concurrency. The old default-using code reports 0 (the built-in default of 2), so these fail on the old code.
  • TestTunedTransport_ReusesIdleConnections: the effect proof. An httptest server with a ConnState callback (set before Start to avoid a race) counts new connections. Two concurrent bursts of 8 requests each run through the tuned client; the second burst reuses the idle pool and opens far fewer than a fresh wave. With the old MaxIdleConnsPerHost = 2 the pool holds only 2, forcing roughly concurrency - 2 re-dials per wave. Deterministic: bounded goroutines, sync.WaitGroup, bodies drained and closed via GetJSON. Ran 30x under -race, no flakes.
  • TestDelete_DrainsAndClosesBody: confirms the Delete body is drained (second read yields nothing) and closed (Close is idempotent).

Validation

go vet clean; CGO_ENABLED=1 go test -race ./... passes; golangci-lint 0 issues; total coverage 69.9% (floor 65%).

Edited by Andrew Dunn

Merge request reports

Loading