api_v4_users fails with more than 7 users
Whilst running GPT against my own 50k environment I noticed the api_v4_users
test was failing. Whilst looking into this I found the reason to be due to the number of users within the system. The environment has gitlab-qa run against it regularly and this leaves some users behind and causes the test to fail.
The below shows the performance hit as i add new users.
4 Users
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
-------------|--------|----------------------|----------|-------------------|----------------|----------------
api_v4_users | 1000/s | 991.33/s (>800.00/s) | 93.20ms | 108.06ms (<200ms) | 100.00% (>99%) | Passed
5 Users
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
-------------|--------|----------------------|----------|-------------------|----------------|----------------
api_v4_users | 1000/s | 990.15/s (>800.00/s) | 111.64ms | 131.21ms (<200ms) | 100.00% (>99%) | Passed
6 Users
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
-------------|--------|----------------------|----------|-------------------|----------------|----------------
api_v4_users | 1000/s | 988.03/s (>800.00/s) | 132.51ms | 157.32ms (<200ms) | 100.00% (>99%) | Passed
7 Users
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
-------------|--------|----------------------|----------|-------------------|----------------|----------------
api_v4_users | 1000/s | 986.38/s (>800.00/s) | 155.36ms | 188.93ms (<200ms) | 100.00% (>99%) | Passed
8 Users
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
-------------|--------|----------------------|----------|-------------------|----------------|-----------------
api_v4_users | 1000/s | 985.69/s (>800.00/s) | 184.57ms | 227.75ms (<200ms) | 100.00% (>99%) | FAILED²
Within GitLab, by default we get 20 results per page when returning multiple results, should GPT be testing the time it takes to return a full page rather than assuming there is only 1 or 2 users?
A quick solution is to add ?per_page=1
to the requests to limit the number of results returned. For now this might be the better solution as I assume this will be the same for all list APIs within GPT. Ideally, in my opinion, GPT should be testing the time it takes to return a full page of users.