Improve performance of Issue Details API to meet target
Summary
The Issue Details API is shown to be unperformant in several performance tests.
On several environments it returns a high TTFB P90 of several seconds. On further analysis it appears the endpoint generally has high Rails CPU usage that can sometimes "trip" over on certainly environment makeups that in turn significantly impacts performance.
In these scenarios with synthetic testing it's more the fact that the endpoint has high CPU usage and is close to "tripping" on the edge. This is much more common on real environments as there's other processes going on so the task here is to optimise the endpoint generally to give it much more headroom.
Additionally this appears to have gotten worse recently, over the last month we've seen the endpoint trip more suggesting something may have changed recently.
Details
Across our test environments we see times ranging from 300ms to 2400ms. On one of the most impact environments, 25k, it posts a time of 2400ms and the metrics show it's it's rails processing the data that's struggling:
* Environment: 25k
* Environment Version: 16.1.0-pre `87fd9f75025`
* Option: 60s_500rps
* Date: 2023-05-22
* Run Time: 1h 39m 11.67s (Start: 04:48:16 UTC, End: 06:27:27 UTC)
* GPT Version: v2.12.2
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|-------|----------------------|-----------|----------------------|----------------|---------
api_v4_projects_issues_issue | 500/s | 406.78/s (>400.00/s) | 1106.67ms | 2342.43ms (<1500ms) | 100.00% (>99%) | FAILED¹²
Note that 80% here is the max for the Puma workers on the box as selected automatically by Omnibus, some cores are left for other processes.
Conversely on one of the environments where it performs better, 10k, it's still taking up a lot of CPU but just not as much:
* Environment: 10k
* Environment Version: 16.1.0-pre `87fd9f75025`
* Option: 60s_200rps
* Date: 2023-05-25
* Run Time: 1h 38m 22.08s (Start: 04:44:50 UTC, End: 06:23:12 UTC)
* GPT Version: v2.12.2
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|-------|----------------------|-----------|-----------------------|----------------|--------
api_v4_projects_issues_issue | 200/s | 192.05/s (>160.00/s) | 307.78ms | 357.74ms (<1500ms) | 100.00% (>99%) | Passed¹
Typically this would be a severity2 based on our performance targets but seeing it's more of a nuanced case I'll mark as a severity3