Keep acceptance test ports unique and fail fast on a failed start
What does this MR do?
This is a follow-up to !1282 (merged). Since that MR moved the acceptance tests to ephemeral ports, tests-acceptance fails now and then with bind: address already in use and then hangs until the 10-minute package timeout. Two things in test/acceptance/helpers_test.go combine to do that.
Two listeners of one launch can be given the same port. allocateFreePort listens on :0, reads the port and closes the listener. Once it is closed the kernel treats the port as free and can return it from the next call, so assignFreePorts can hand one port to two listener roles of the same launch. The comment there says collisions are effectively impossible, but the pool is smaller than it looks: every port in the CI logs below is odd and below 46884. That is how Linux serves a :0 bind on a socket with SO_REUSEADDR, which Go sets on listeners: inet_csk_find_open_port searches the odd ports of the lower half of ip_local_port_range first, about 7000 ports with the default range. On master, 25 of 20000 assignFreePorts(t, supportedListeners()) calls gave one port to two roles, about one launch in 800 (five roles, with IPv6 available; the CI launches below have four). Two earlier runs gave 27 and 31.
A failed start hangs instead of failing. The goroutine waiting on the process calls require.NoError(t, cmd.Wait()). When the process exits non-zero, FailNow stops that goroutine before it reaches close(waitCh), so WaitUntilRequestSucceeds runs out its 5 seconds, cleanup then blocks on <-waitCh for good, and the package only ends at panic: test timed out after 10m0s. Before !1282 (merged), cleanup failed on Signal for a process that had already exited and never reached <-waitCh, so a failed bind failed its test after 5 seconds; !1282 (merged) made cleanup accept os.ErrProcessDone and wait, which turns the goroutine's assertion into a hang.
What I changed, in test/acceptance:
allocateFreePortreserves each port it hands out until the test that asked for it finishes, and probes again when the kernel returns a reserved port. The reservation is released throught.Cleanup, which runs after the cleanupRunPagesProcessregisters has stopped the process. Every allocation in the package goes through this function (assignFreePortsand the three config tests that setLISTEN_HTTPthemselves), so nothing else needed to change. The window between closing the probe and the binary binding the port is still there for other processes, as before.- The fixed ports some tests use start reserved: the metrics address 42345 that
TestPrometheusMetricsCanBeScraped,TestMetricsHTTPSConnectionand the TLS rate limit tests pass, and the object storage stub's 38001. Both are odd and below 46884, so the kernel hands them out like any other port, and a launch that gave 42345 to a listener while also passing it as-metrics-addresswould fail to start the same way. The stub binds 38001 before its tests allocate anything, so reserving that one is a precaution. Both ports are the constantsmetricsPortandobjectStoragePortinhelpers_test.go, which the reservation is built from and the tests use, so the reservation cannot fall behind a changed port. - The waiting goroutine stores the error from
cmd.Wait()and closeswaitChin adefer, andcleanupchecks that error on the test goroutine after<-waitCh. A process that fails to start now fails its test as soon as it exits:WaitUntilRequestSucceedsseeswaitChclosed and returns, andcleanupreports the exit status (exit status 1) on the test goroutine. The rest of the package keeps running.
It is a test-only change, so the commits have no Changelog trailer. I did not add the checks below as tests: the duplicate is probabilistic, and the hang reproduction is a test that fails on purpose.
CI jobs with the same port twice in one launch
Each of these logs failed to listen on addr 127.0.0.1:<port>: bind: address already in use and then panic: test timed out after 10m0s; the jobs ran for 11 to 12 minutes.
| Job | Ref | Test | Port | Given to |
|---|---|---|---|---|
| 17019638534 | refs/merge-requests/1344/merge |
TestQueryStringPersistedInSlashRewrite |
38311 | -listen-http, -listen-https-proxyv2 |
| 16959423791 | refs/merge-requests/1341/train |
TestDiskDisabledFailsToServeFileAndZipContent |
34739 | -listen-https, -listen-https-proxyv2 |
| 17001552186 | refs/merge-requests/1339/merge |
TestGroupDomainResponse200 |
44993 | -listen-https, -listen-proxy |
| 15943318539 | master |
TestPrometheusMetricsCanBeScraped |
38545 | -listen-https, -listen-proxy |
Of the 42 tests-acceptance jobs that have failed since !1282 (merged) merged (counted on 2026-10-10), 14 failed this way, each with one port twice in the arguments of a single launch.
Before and after, locally
Scratch tests I ran against master (811984e6) and this branch, in golang:1.27.1 on Linux 6.12 with the default ip_local_port_range of 32768 60999 and IPv6 available. They are not part of this MR.
| Check | master | this branch |
|---|---|---|
20000 calls of assignFreePorts(t, supportedListeners()), each in its own subtest |
25 gave one port to two roles | 0 |
50000 subtests calling allocateFreePort twice, as TestMultipleListenersFromEnvironmentVariables does |
8 equal pairs | 0 |
The same 50000: a LISTEN_HTTP port, then assignFreePorts for httpsListener, as TestMixedConfigSources does |
1 time the same port | 0 |
50000 subtests calling allocateFreePort on 127.0.0.1 and on ::1 |
42345 handed out 5 and 5 times, 38001 8 and 6 times | neither handed out |
gitlab-pages started with an extra listener on a port the test holds, -timeout 60s |
blocks in cleanup at <-waitCh, panic: test timed out after 1m0s, the next test never runs |
fails in 0.07s with exit status 1, the next test passes |
The hang reproduction:
func TestScratchFailedBindEndsTheTest(t *testing.T) {
held, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
defer held.Close()
RunPagesProcess(t,
withListeners([]ListenSpec{httpListener}),
withExtraArgument("listen-http", held.Addr().String()),
)
}On this branch:
go build -o gitlab-pages . && go test ./test/acceptance/ -count=1: ok withgolang:1.27.1andgolang:1.26.8go test -race ./test/acceptance/ -count=1: ok, no data racesgo test ./test/acceptance/ -shuffle=on -count=3: okgolangci-lint run ./...ingolangci/golangci-lint:v2.12.2, ascode_qualityruns it: 0 issuesgo vet ./test/acceptance/andgofmt -l test/acceptance/: clean
Related to !1282 (merged).
TODO
- Feature flag
- Added feature flag:
- This feature does not require a feature flag
- I added the
Changelogtrailer to the commits that need to be included in the changelog (e.g.Changelog: added): test-only change, no changelog entry - I added unit tests or they are not required
- I added acceptance tests or they are not required
- I added documentation (or it's not required)
- I followed code review guidelines
- I followed Go Style guidelines