Intermittent deadlock in TestRemoveRepository_removeReplicationEvents
Test TestRemoveRepository_removeReplicationEvents
failed to complete in https://gitlab.com/gitlab-org/gitaly/-/jobs/2554792054.
Ticker appears to have blocked writing to its output channel.
- https://gitlab.com/gitlab-org/gitaly/-/blob/70d6aa021ebfc05d9d727a7eb4c9ff4782db4c30/cmd/praefect/subcmd_remove_repository_test.go#L327
- https://gitlab.com/gitlab-org/gitaly/-/blob/70d6aa021ebfc05d9d727a7eb4c9ff4782db4c30/internal/helper/ticker.go#L60
Unable to reproduce running this test 1000x locally.
Panic stack traces:
panic: test timed out after 10m0s
goroutine 2098 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:1788 +0xbb
created by time.goFunc
/usr/local/go/src/time/sleep.go:180 +0x4a
goroutine 1 [chan receive, 9 minutes]:
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1225 +0x635
testing.tRunner(0xc0037c4680, 0xc0038df9a8)
/usr/local/go/src/testing/testing.go:1265 +0x269
testing.runTests(0xc0003d1a00, {0x3a40e80, 0x14, 0x14}, {0xc0038dfd38, 0x0, 0x3a57520})
/usr/local/go/src/testing/testing.go:1596 +0x7cb
testing.(*M).Run(0xc0003d1a00)
/usr/local/go/src/testing/testing.go:1504 +0x9d2
gitlab.com/gitlab-org/gitaly/v15/internal/testhelper.Run.func1({0x0, 0x0, 0x3a582a0}, 0xc0003d1a00)
/builds/gitlab-org/gitaly/internal/testhelper/configure.go:71 +0x377
gitlab.com/gitlab-org/gitaly/v15/internal/testhelper.Run(0x46fce5, {0x0, 0x0, 0x0})
/builds/gitlab-org/gitaly/internal/testhelper/configure.go:74 +0x5b
gitlab.com/gitlab-org/gitaly/v15/cmd/praefect.TestMain(...)
/builds/gitlab-org/gitaly/cmd/praefect/main_test.go:21
main.main()
_testmain.go:83 +0x269
goroutine 118 [chan send, 9 minutes]:
gitlab.com/gitlab-org/gitaly/v15/internal/helper.(*ManualTicker).Tick(...)
/builds/gitlab-org/gitaly/internal/helper/ticker.go:60
gitlab.com/gitlab-org/gitaly/v15/cmd/praefect.TestRemoveRepository_removeReplicationEvents(0xc0039cab60)
/builds/gitlab-org/gitaly/cmd/praefect/subcmd_remove_repository_test.go:327 +0x1165
testing.tRunner(0xc0039cab60, 0x209deb8)
/usr/local/go/src/testing/testing.go:1259 +0x230
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1306 +0x727
goroutine 36 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000522000)
/builds/gitlab-org/gitaly/.go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0x185
created by go.opencensus.io/stats/view.init.0
/builds/gitlab-org/gitaly/.go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0xf9
goroutine 11 [chan receive, 10 minutes]:
gitlab.com/gitlab-org/gitaly/v15/internal/gitaly/rubyserver/balancer.(*builder).monitor(0xc0037b98f0)
/builds/gitlab-org/gitaly/internal/gitaly/rubyserver/balancer/balancer.go:143 +0xcd
created by gitlab.com/gitlab-org/gitaly/v15/internal/gitaly/rubyserver/balancer.newBuilder
/builds/gitlab-org/gitaly/internal/gitaly/rubyserver/balancer/balancer.go:120 +0x219
goroutine 1017 [select, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc0029ad040, {0x2ae6f90, 0xc0037c8c00})
/usr/local/go/src/database/sql/sql.go:1196 +0xfa
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:794 +0x307