Improve performance of action for Projects::MergeRequestsController under load into S4 tier on larger environments
The Merge Request Discussions API had it's performance recently improved notably but recent testing has found it's still unperformant on our larger Reference Architectures (specifically the 25k and https://docs.gitlab.com/ee/administration/reference_architectures/50k_users.html) where the load is higher.
25k:
* Environment: 25k
* Environment Version: 13.1.0-pre `89b3e1dcda8`
* Option: 60s_500rps
* Date: 2020-06-08
* Run Time: 1h 4m 33.59s (Start: 01:28:32 UTC, End: 02:33:05 UTC)
* GPT Version: v2.0.0
❯ Overall Results Score: 95.46%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|-------|----------------------|------------|-----------------------|----------------|--------
api_v4_projects_merge_requests_merge_request_discussions | 500/s | 400.22/s (>400.00/s) | 1126.51ms | 1308.01ms (<500ms) | 100.00% (>95%) | FAILED²
50k:
* Environment: 50k
* Environment Version: 13.1.0-pre `839350e271d`
* Option: 60s_1000rps
* Date: 2020-06-10
* Run Time: 2m 49.43s (Start: 14:44:04 UTC, End: 14:46:53 UTC)
* GPT Version: v2.0.0
❯ Overall Results Score: 87.65%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|--------|----------------------|-----------|--------------------|----------------|-----------------
api_v4_projects_merge_requests_merge_request_discussions | 1000/s | 876.45/s (>800.00/s) | 1002.88ms | 1328.93ms (<500ms) | 100.00% (>95%) | FAILED²
Unlike the last issue there's no obvious bottlenecks that can be found for this in terms of resources. It's only seen on these larger environments with smaller environments posting results comfortably under the 500ms target.
As part of the new performance targets this endpoint is in our ~S3 tier for these environments.
Task is to reduce the performance of this endpoint on these larger environment into the ~s4 tier .