Skip to content

limithandler: Fix flaky UnaryLimitHandler test

Patrick Steinhardt requested to merge pks-limithandler-fix-flakiness-in-unary-interceptor-test into master

One of our tests for the UnaryLimitHandler is regularly flaking with the following error:

=== FAIL: internal/grpc/middleware/limithandler TestUnaryLimitHandler (0.12s)
    middleware_test.go:73:
                Error Trace:	/builds/gitlab-org/gitaly/internal/grpc/middleware/limithandler/middleware_test.go:73
                Error:      	Not equal:
                                expected: 2
                                actual  : 0
                Test:       	TestUnaryLimitHandler

This happens especially in the case where the test machine is heavily loaded. What happens here is that we first queue a bunch of RPC calls via Goroutines, wait a 100 milliseconds for them to settle, and then verify that we got exactly two requests. The intent is to verify that we don't get more than two requests due to the concurrency limit we put in place, but when the system is loaded it can also happen that we get less than these requests if the Goroutines weren't yet scheduled.

Fix this by converting the test to use the queueing test server. Like this, we can await the first two incoming requests synchronously and thus know that we have hit the concurrency limit. We can then do a best effort wait for additional requests, which shouldn't happen.

Merge request reports