Merge Request List API slows notably when number of MRs are higher
Summary
Performance testing has found the Merge Requests List API has slow performance when there's a high amount of MRs (~3500 in our test data). On a target 10k reference environment testing has found the endpoint to typically take 3 seconds to return, which is considered too high. It also degrades further in larger environments such as our in development 25k environment where it returned in 4.5 seconds.
Most endpoints return under 500ms so any that stand out like this are to be raised and investigated.
Recent results from our 10k environment over several previous versions:
12.3.2
Environment: 10k (12.3.2-ee 9ae54250d64)
Scenario: 60s_200rps
Date: 2019-10-01
Run Time: 1645.64s (Start: 13:12:50 UTC, End: 13:40:15 UTC)
NAME | DURATION | P95 | RPS | SUCCESS RATE | RESULT
---------------------------------------------------------|----------|------------|----------------------|----------------|-------
api_v4_projects_merge_requests | 60.0s | 3083.84ms | 88.37/s (>40.00/s) | 100.00% (>95%) | Passed
12.2.4:
Environment: 10k (12.2.4-ee e57d66ff40e)
Scenario: 60s_200rps
Date: 2019-09-26
Run Time: 1646.34s (Start: 01:20:54 UTC, End: 01:48:21 UTC)
NAME | DURATION | P95 | RPS | RPS_THRESHOLD | RESULT
---------------------------------------------------------|----------|------------|----------------------|-----------------|-------
api_v4_projects_merge_requests | 60.0s | 3102.49ms | 77.68313/s (4661) | 56.00/s (3360) | Passed
12.1.6:
Environment: 10k (12.1.6-ee d05ee0a9c12)
Scenario: 60s_200rps
Date: 2019-08-26
Run Time: 1558.1s (Start: 01:20:11 UTC, End: 01:46:10 UTC)
NAME | RESULT | DURATION | P95 | RPS_COUNT | RPS_MEAN
-----------------------------------------------------|--------|----------|------------|-----------|-------------
api_v4_projects_merge_requests | Passed | 60.0s | 3936.42ms | 4571 | 76.183245/s
12.0.2:
Environment: 10k (12.0.2-ee ef76b54fc1e)
Scenario: 60s_200rps
Date: 2019-08-20
Run Time: 1551.87s (Start: 01:20:08 UTC, End: 01:46:00 UTC)
NAME | RESULT | DURATION | P95 | RPS_COUNT | RPS_MEAN
-----------------------------------------------------|--------|----------|------------|-----------|-------------
api_v4_projects_merge_requests | Passed | 60.0s | 3237.82ms | 4617 | 76.949889/s
Finally to show the endpoint degrades even further at higher throughputs the following is a result from our target 25k environment:
12.3.2
Environment: 25k (12.3.2-ee 9ae54250d64)
Scenario: 60s_500rps
Date: 2019-10-01
Run Time: 1879.28s (Start: 16:34:52 UTC, End: 17:06:11 UTC)
NAME | DURATION | P95 | RPS | SUCCESS RATE | RESULT
---------------------------------------------------------|----------|------------|----------------------|----------------|-------
api_v4_projects_merge_requests | 60.0s | 4556.35ms | 129.92/s (>100.00/s) | 100.00% (>95%) | Passed
Steps to reproduce
- Check out the Performance Toolkit
- Run the specific test with the
run-k6
command. For example against the 10k environment you would run this following from the project root:./run-k6 -e environments/10k.json -s scenarios/60s_200rps.json -t tests/api/api_v4_projects_merge_requests.js
. You will need an ACCESS_TOKEN for this endpoint as well - also covered in docs but give me a shout for the token to use against our 10k environment. - If you're seeking to run the test against your own environment the Toolkit's documentation has details on how to achieve this.
What is the current bug behavior?
The API is slow with higher amounts of MRs (~3500). Testing has found it to be around 3 seconds on our certified 10k architecture.
What is the expected correct behavior?
That the API responds inline with other endpoints performance, typically under 500ms.