Investigate large difference with Single Issue API test on 25k
We now test the Issue Details API with GPT, which comes in typically under the performance target of 500ms.
However, on our 25k reference architecture we're seeing a completely different result with it coming in around 9-10s:
* Environment: 25k
* Environment Version: 12.9.0-pre `062dcdf9199`
* Option: 60s_500rps
* Date: 2020-03-17
* Run Time: 1m 13.24s (Start: 10:27:25 UTC, End: 10:28:38 UTC)
* GPT Version: v1.2.4
➤ Overall Results Score: 10.66%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
-----------------------------|-------|---------------------|-----------|---------------------|---------------|-----------------
api_v4_projects_issues_issue | 500/s | 53.35/s (>400.00/s) | 7932.48ms | 10020.15ms (<500ms) | 99.87% (>95%) | FAILED³
This is a massive increase and difference to what we're seeing on the other architectures, including the 50k.
Initial analysis shows the gitlab-rails nodes are completely maxed during this test:

- 25k full metrics: https://snapshot.raintank.io/dashboard/snapshot/r0C6DXyzgALa5MGoToX2Oj3LvXe83MOC
- 50k full metrics: https://snapshot.raintank.io/dashboard/snapshot/Eb0qpQmTytB0JcXsJH5FUtxvJoqTPwSi
Again this is not the case for every other environment, including 50k, which shows the rails nodes showing around 40% usage and postgres showing notably higher.
Task is to investigate why this is happening.
cc @niskhakova
Edited by Grant Young