Gitaly N+1 Detector Tests
Context: https://gitlab.slack.com/archives/C3ER3TQBT/p1549395473729500
[The Gitaly N+1 detector is] a tool we used during the gitaly migration project to get a grip on the number of gitaly requests made per rails request. The current limit is an arbitrary number that seemed OK at the time. We could make an effort of driving it down.
That effort has begun here: gitaly#1504
The number of gitaly calls made depends on the data in the system (as well as the way the code is written). E.g., more repos/projects/MRs/etc means potentially more calls. Therefore, there's value in running the tests in an environment with more data.
- We can have a CI job which runs the test with Gitlay N+1 limit lowered. So the steps could be something like this:
On Staging (or in the performance testing environment):
- Have
lower_n_plus_1_limit
flag turned ON - (Hoping this can be done using a feature flag - which we can turn on and off? ) - Run the E2E tests - see any failures that happen.
- Report the failures - (Update/Create an issue?)
- Turn
lower_n_plus_1_limit
flag OFF.
To do that we'd need to:
- modify gitaly_client.rb to allow the limit to be:
-
enabled via a feature flag since it's not enable at all in production or production-like environments (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25931) - [-] lowered via a feature flag https://gitlab.com/gitlab-org/gitlab-ce/issues/61538
-
[ ] (and optionally to allow the limit to be set via some other means, e.g., an environment variable)Not for the first iteration
-
-
allow the QA test framework to toggle the feature flag on/off: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26060 -
add a pipeline that runs all the e2e tests with the flag on. https://gitlab.com/gitlab-org/gitlab-ce/issues/58905 - [-] consider when to schedule the tests if they're run in a shared environment (https://gitlab.com/gitlab-org/gitlab-ce/issues/60770)
Edited by Mark Lapierre