Flaky leaked goroutine from DNS lookup of tcp://non-existing
This appears to have been introduced with 96af8f1c, which adds a test that attempts to connect to tcp://non-existing:42
. The DNS lookup for this address intermittently takes longer to complete that the packages' tests, causing the unexpected goroutine error.
On my own host Praefect is waiting for a response from io.systemd.Resolve.ResolveHostname
when the tests complete.
{"method":"io.systemd.Resolve.ResolveHostname","parameters":{"name":"non-existing","flags":0}
Example failure: https://gitlab.com/gitlab-org/gitaly/-/pipelines/671096845
Backtrace:
panic: goroutines running: found unexpected goroutines:
[Goroutine 97 in state chan receive, with net.(*Resolver).goLookupIPCNAMEOrder.func4 on top of the stack:
goroutine 97 [chan receive]:
net.(*Resolver).goLookupIPCNAMEOrder.func4({_, _}, _)
/usr/local/go/src/net/dnsclient_unix.go:618 +0x85
net.(*Resolver).goLookupIPCNAMEOrder(_, {_, _}, {_, _}, {_, _}, _)
/usr/local/go/src/net/dnsclient_unix.go:628 +0x822
net.(*Resolver).lookupIP(0x3224260, {0x1f0b0f8, 0xc001ee3ec0}, {0x18002c9, 0x3}, {0xc001ef8b6f, 0xc})
/usr/local/go/src/net/lookup_unix.go:102 +0x172
net.glob..func1({0x1f0b0f8?, 0xc001ee3ec0?}, 0x511ce6?, {0x18002c9?, 0xc00006e784?}, {0xc001ef8b6f?, 0xc00006e798?})
/usr/local/go/src/net/hook.go:23 +0x3d
net.(*Resolver).lookupIPAddr.func1()
/usr/local/go/src/net/lookup.go:319 +0x9f
internal/singleflight.(*Group).doCall(0x3224270, 0xc003055a90, {0xc001bf9750, 0x10}, 0xc001c84080?)
/usr/local/go/src/internal/singleflight/singleflight.go:95 +0x3b
created by internal/singleflight.(*Group).DoChan
/usr/local/go/src/internal/singleflight/singleflight.go:88 +0x2ec
Goroutine 114 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
goroutine 114 [IO wait]:
internal/poll.runtime_pollWait(0x7f91e7217ba8, 0x72)
/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0003c3f80?, 0xc00305d900?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0003c3f80, {0xc00305d900, 0x4d0, 0x4d0})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc0003c3f80, {0xc00305d900?, 0xc001c1a3c0?, 0x452152?})
/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0008a0cc0, {0xc00305d900?, 0x621452?, 0xc0003c3f80?})
/usr/local/go/src/net/net.go:183 +0x45
net.dnsPacketRoundTrip({_, _}, _, {{{0x6e, 0x6f, 0x6e, 0x2d, 0x65, 0x78, 0x69, ...}, ...}, ...}, ...)
/usr/local/go/src/net/dnsclient_unix.go:91 +0x19d
net.(*Resolver).exchange(_, {_, _}, {_, _}, {{{0x6e, 0x6f, 0x6e, 0x2d, 0x65, ...}, ...}, ...}, ...)
/usr/local/go/src/net/dnsclient_unix.go:170 +0x608
net.(*Resolver).tryOneName(_, {_, _}, _, {_, _}, _)
/usr/local/go/src/net/dnsclient_unix.go:260 +0x485
net.(*Resolver).goLookupIPCNAMEOrder.func3.1(0x1?)
/usr/local/go/src/net/dnsclient_unix.go:612 +0x85
created by net.(*Resolver).goLookupIPCNAMEOrder.func3
/usr/local/go/src/net/dnsclient_unix.go:611 +0x16c
]
goroutine 1 [running]:
gitlab.com/gitlab-org/gitaly/v15/internal/testhelper.mustHaveNoGoroutines()
/builds/gitlab-org/gitaly/internal/testhelper/leakage.go:36 +0x34e
gitlab.com/gitlab-org/gitaly/v15/internal/testhelper.Run.func1({0x0, 0x0, 0x0?}, 0x189830f?)
/builds/gitlab-org/gitaly/internal/testhelper/configure.go:73 +0x295
gitlab.com/gitlab-org/gitaly/v15/internal/testhelper.Run(0x0?, {0x0?, 0x6?, 0xc000072710?})
/builds/gitlab-org/gitaly/internal/testhelper/configure.go:74 +0x31
gitlab.com/gitlab-org/gitaly/v15/internal/praefect/service/server_test.TestMain(...)
/builds/gitlab-org/gitaly/internal/praefect/service/server/testhelper_test.go:10
main.main()
_testmain.go:109 +0x297
FAIL gitlab.com/gitlab-org/gitaly/v15/internal/praefect/service/server 4.355s
Reproduction Steps
- open wifi network settings on your OS, navigate to DNS tab and enter a random IP which you know isn’t a DNS server e.g.
192.0.0.0
- run tests
CGO_ENABLED=0 go test --run TestServer_ReadinessCheck_unreachableGitaly ./internal/praefect/service/server -v --count 1
CGO_ENABLED=1 go test --run TestServer_ReadinessCheck_unreachableGitaly ./internal/praefect/service/server -v --count 1
Edited by John McDonnell