Keep acceptance test ports unique and fail fast on a failed start

What does this MR do?

This is a follow-up to !1282 (merged). Since that MR moved the acceptance tests to ephemeral ports, tests-acceptance fails now and then with bind: address already in use and then hangs until the 10-minute package timeout. Two things in test/acceptance/helpers_test.go combine to do that.

Two listeners of one launch can be given the same port. allocateFreePort listens on :0, reads the port and closes the listener. Once it is closed the kernel treats the port as free and can return it from the next call, so assignFreePorts can hand one port to two listener roles of the same launch. The comment there says collisions are effectively impossible, but the pool is smaller than it looks: every port in the CI logs below is odd and below 46884. That is how Linux serves a :0 bind on a socket with SO_REUSEADDR, which Go sets on listeners: inet_csk_find_open_port searches the odd ports of the lower half of ip_local_port_range first, about 7000 ports with the default range. On master, 25 of 20000 assignFreePorts(t, supportedListeners()) calls gave one port to two roles, about one launch in 800 (five roles, with IPv6 available; the CI launches below have four). Two earlier runs gave 27 and 31.

A failed start hangs instead of failing. The goroutine waiting on the process calls require.NoError(t, cmd.Wait()). When the process exits non-zero, FailNow stops that goroutine before it reaches close(waitCh), so WaitUntilRequestSucceeds runs out its 5 seconds, cleanup then blocks on <-waitCh for good, and the package only ends at panic: test timed out after 10m0s. Before !1282 (merged), cleanup failed on Signal for a process that had already exited and never reached <-waitCh, so a failed bind failed its test after 5 seconds; !1282 (merged) made cleanup accept os.ErrProcessDone and wait, which turns the goroutine's assertion into a hang.

What I changed, in test/acceptance:

  • allocateFreePort reserves each port it hands out until the test that asked for it finishes, and probes again when the kernel returns a reserved port. The reservation is released through t.Cleanup, which runs after the cleanup RunPagesProcess registers has stopped the process. Every allocation in the package goes through this function (assignFreePorts and the three config tests that set LISTEN_HTTP themselves), so nothing else needed to change. The window between closing the probe and the binary binding the port is still there for other processes, as before.
  • The fixed ports some tests use start reserved: the metrics address 42345 that TestPrometheusMetricsCanBeScraped, TestMetricsHTTPSConnection and the TLS rate limit tests pass, and the object storage stub's 38001. Both are odd and below 46884, so the kernel hands them out like any other port, and a launch that gave 42345 to a listener while also passing it as -metrics-address would fail to start the same way. The stub binds 38001 before its tests allocate anything, so reserving that one is a precaution. Both ports are the constants metricsPort and objectStoragePort in helpers_test.go, which the reservation is built from and the tests use, so the reservation cannot fall behind a changed port.
  • The waiting goroutine stores the error from cmd.Wait() and closes waitCh in a defer, and cleanup checks that error on the test goroutine after <-waitCh. A process that fails to start now fails its test as soon as it exits: WaitUntilRequestSucceeds sees waitCh closed and returns, and cleanup reports the exit status (exit status 1) on the test goroutine. The rest of the package keeps running.

It is a test-only change, so the commits have no Changelog trailer. I did not add the checks below as tests: the duplicate is probabilistic, and the hang reproduction is a test that fails on purpose.

CI jobs with the same port twice in one launch

Each of these logs failed to listen on addr 127.0.0.1:<port>: bind: address already in use and then panic: test timed out after 10m0s; the jobs ran for 11 to 12 minutes.

Job Ref Test Port Given to
17019638534 refs/merge-requests/1344/merge TestQueryStringPersistedInSlashRewrite 38311 -listen-http, -listen-https-proxyv2
16959423791 refs/merge-requests/1341/train TestDiskDisabledFailsToServeFileAndZipContent 34739 -listen-https, -listen-https-proxyv2
17001552186 refs/merge-requests/1339/merge TestGroupDomainResponse200 44993 -listen-https, -listen-proxy
15943318539 master TestPrometheusMetricsCanBeScraped 38545 -listen-https, -listen-proxy

Of the 42 tests-acceptance jobs that have failed since !1282 (merged) merged (counted on 2026-10-10), 14 failed this way, each with one port twice in the arguments of a single launch.

Before and after, locally

Scratch tests I ran against master (811984e6) and this branch, in golang:1.27.1 on Linux 6.12 with the default ip_local_port_range of 32768 60999 and IPv6 available. They are not part of this MR.

Check master this branch
20000 calls of assignFreePorts(t, supportedListeners()), each in its own subtest 25 gave one port to two roles 0
50000 subtests calling allocateFreePort twice, as TestMultipleListenersFromEnvironmentVariables does 8 equal pairs 0
The same 50000: a LISTEN_HTTP port, then assignFreePorts for httpsListener, as TestMixedConfigSources does 1 time the same port 0
50000 subtests calling allocateFreePort on 127.0.0.1 and on ::1 42345 handed out 5 and 5 times, 38001 8 and 6 times neither handed out
gitlab-pages started with an extra listener on a port the test holds, -timeout 60s blocks in cleanup at <-waitCh, panic: test timed out after 1m0s, the next test never runs fails in 0.07s with exit status 1, the next test passes

The hang reproduction:

func TestScratchFailedBindEndsTheTest(t *testing.T) {
	held, err := net.Listen("tcp", "127.0.0.1:0")
	require.NoError(t, err)
	defer held.Close()

	RunPagesProcess(t,
		withListeners([]ListenSpec{httpListener}),
		withExtraArgument("listen-http", held.Addr().String()),
	)
}

On this branch:

  • go build -o gitlab-pages . && go test ./test/acceptance/ -count=1: ok with golang:1.27.1 and golang:1.26.8
  • go test -race ./test/acceptance/ -count=1: ok, no data races
  • go test ./test/acceptance/ -shuffle=on -count=3: ok
  • golangci-lint run ./... in golangci/golangci-lint:v2.12.2, as code_quality runs it: 0 issues
  • go vet ./test/acceptance/ and gofmt -l test/acceptance/: clean

Related to !1282 (merged).

TODO

Edited by José M. Requena Plens

Merge request reports

Loading
Loading