Racy test "TestRemovals/add_then_remove_but_too_fast"

Another new racy test:

=== RUN   TestRemovals/add_then_remove_but_too_fast
    balancer_test.go:176: 
        	Error Trace:	balancer_test.go:176
        	Error:      	Not equal: 
        	            	expected: false
        	            	actual  : true
        	Test:       	TestRemovals/add_then_remove_but_too_fast
        	Messages:   	expected result from removing "foo"
=== RUN   TestRemovals/remove_one_address_too_many
=== RUN   TestRemovals/remove_unknown_address
=== RUN   TestRemovals/remove_standby_address
--- FAIL: TestRemovals (0.02s)
    --- PASS: TestRemovals/add_then_remove (0.01s)
    --- FAIL: TestRemovals/add_then_remove_but_too_fast (0.00s)
    --- PASS: TestRemovals/remove_one_address_too_many (0.01s)
    --- PASS: TestRemovals/remove_unknown_address (0.00s)
    --- PASS: TestRemovals/remove_standby_address (0.00s)
FAIL
FAIL	gitlab.com/gitlab-org/gitaly/internal/gitaly/rubyserver/balancer	0.045s