Flaky test in CI: TestRemovals in balancer_test.go

I've noticed that on CI, a balancer test is flaky, in for example: https://gitlab.com/gitlab-org/gitaly/-/jobs/165930059

--- FAIL: TestRemovals (0.03s)
    --- FAIL: TestRemovals/add_then_remove_but_too_fast (0.00s)
        <autogenerated>:1: 
            	Error Trace:	balancer_test.go:179
            	Error:      	Not equal: 
            	            	expected: true
            	            	actual  : false
            	Test:       	TestRemovals/add_then_remove_but_too_fast
            	Messages:   	expected result from removing "bar"
FAIL
FAIL	gitlab.com/gitlab-org/gitaly/internal/rubyserver/balancer	0.078s

Tried to reproduce locally using go test -race -count=1000 -run=TestRemovals ./internal/rubyserver/balancer but couldn't reproduce.