Investigate sporadic high Postgres CPU usage on 50k

When reviewing some test results for the larger reference architectures (25k and more specifically 50k) there's now several failures happening that look to be due to the Postgres node being maxed out.

On a fresh 50k environment the following failures occurred:

* Environment:                50k
* Environment Version:        13.1.0-pre `839350e271d`
* Option:                     60s_1000rps
* Date:                       2020-06-09
* Run Time:                   1h 21m 12.03s (Start: 12:44:57 UTC, End: 14:06:09 UTC)
* GPT Version:                v2.0.0

NAME                                                     | RPS    | RPS RESULT           | TTFB AVG   | TTFB P90              | REQ STATUS     | RESULT 
---------------------------------------------------------|--------|----------------------|------------|-----------------------|----------------|--------
api_v4_projects_merge_requests_merge_request             | 1000/s | 827.6/s (>800.00/s)  | 1091.94ms  | 1436.45ms (<500ms)    | 99.74% (>95%)  | FAILED²
api_v4_projects_merge_requests_merge_request_changes     | 1000/s | 945.4/s (>800.00/s)  | 919.69ms   | 1371.24ms (<500ms)    | 86.48% (>95%)  | FAILED²
api_v4_projects_merge_requests_merge_request_discussions | 1000/s | 833.48/s (>800.00/s) | 1049.58ms  | 1404.60ms (<500ms)    | 100.00% (>95%) | FAILED²
api_v4_projects_project                                  | 1000/s | 877.23/s (>800.00/s) | 1018.24ms  | 1349.48ms (<500ms)    | 100.00% (>95%) | FAILED²

When reviewing the metrics it was clear Postgres looked to be outright under-specced CPU wise:

Screenshot_2020-06-09_at_15.40.15

Note that while only 4 tests actually failed numerous others appear to be spiking Postgres near to 100% suggesting they would be close to failing also.

Task is to investigate the increase in CPU usage.

Edited by Grant Young