Improve performance of Merge Request Details API under load into next tier
Following a review done in conjunction with the groupcode review team a new target MR was selected for performance testing that was more realistic and representative of the 99th percentile of all MRs on .com.
When testing against this new MR the Merge Request Details API returned a notably slower TTFB P90 performance time of around 600ms compared to the one before regularly returning times around 150ms.
* Environment: 2k
* Environment Version: 13.12.0-pre `2ae25491ab4`
* Option: 60s_40rps
* Date: 2021-05-20
* Run Time: 1h 25m 13.76s (Start: 04:28:10 UTC, End: 05:53:24 UTC)
* GPT Version: v2.8.0
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|------|--------------------|------------|-----------------------|----------------|--------
api_v4_projects_merge_requests_merge_request | 40/s | 39.35/s (>32.00/s) | 332.90ms | 517.59ms (<500ms) | 100.00% (>99%) | FAILED²
Looking at the environment metrics the endpoint appears to be hitting both Rails and Postgres equally with a heavy load:
This suggests there's still performance deficiencies with the API depending on what the MR contains.
Testing was done on our test 10k Reference Architecture environment as standard at 200 RPS with the project being tested a copy of gitlabhq (tarball can be found here). A copy of the specific MR being tested can be found on Staging here.
As per our performance targets the API is now above our main target of 200ms and falls under the severity4 tier. Task is to improve the API's performance to meet the target.