Improve performance of diffs_batch and diffs_metadata actions for Projects::MergeRequestsController under load into S3 tier

Related #30507 (closed)

Performance testing is showing that the diffs_batch.json and diffs_metadata.json actions for Projects::MergeRequestsController are both over our target of 500ms under load respectively. Both are found on the Merge Requests page, specifically the Changes (or Diffs) tab:

 █ Results summary
 * Environment:                10k
 * Environment Version:        12.9.0-pre `545f4a5af48`
 * Option:                     60s_200rps
 * Date:                       2020-03-09
 * Run Time:                   47m 28.13s (Start: 04:28:41 UTC, End: 05:16:09 UTC)
 * GPT Version:                v1.2.2

 NAME                                                     | RPS   | RPS RESULT           | TTFB AVG  | TTFB P90             | REQ STATUS     | RESULT
 ---------------------------------------------------------|-------|----------------------|-----------|----------------------|----------------|-------
 web_project_merge_request_changes                        | 20/s  | 12.27/s (>8.00/s)    | 1749.83ms | 2467.22ms (<4000ms)  | 100.00% (>95%) | Passed

Detailed Stats:
     █ Web - Merge Request Changes Page
     data_received........................: 493 MB  8.2 MB/s
     data_sent............................: 189 kB  3.2 kB/s
     group_duration.......................: avg=10625.41ms min=9167.87ms med=10515.09ms max=13304.80ms p(90)=11799.86ms p(95)=12374.46ms
     http_req_blocked.....................: avg=0.22ms     min=0.00ms    med=0.01ms     max=19.70ms    p(90)=0.88ms     p(95)=0.92ms    
     http_req_connecting..................: avg=0.17ms     min=0.00ms    med=0.00ms     max=9.17ms     p(90)=0.78ms     p(95)=0.82ms    
     http_req_duration....................: avg=1757.60ms  min=162.03ms  med=1888.76ms  max=4133.90ms  p(90)=2469.87ms  p(95)=2675.53ms 
     http_req_receiving...................: avg=7.72ms     min=0.15ms    med=6.25ms     max=118.23ms   p(90)=13.00ms    p(95)=19.91ms   
     http_req_sending.....................: avg=0.05ms     min=0.02ms    med=0.04ms     max=0.18ms     p(90)=0.09ms     p(95)=0.11ms    
     http_req_tls_handshaking.............: avg=0.00ms     min=0.00ms    med=0.00ms     max=0.00ms     p(90)=0.00ms     p(95)=0.00ms    
     http_req_waiting.....................: avg=1749.83ms  min=161.82ms  med=1880.52ms  max=4126.18ms  p(90)=2467.22ms  p(95)=2656.65ms 
     ✓ { endpoint:diffs_batch.json }......: avg=1951.29ms  min=1407.50ms med=1882.63ms  max=4126.18ms  p(90)=2439.88ms  p(95)=2628.55ms 
     ✓ { endpoint:diffs_metadata.json }...: avg=2294.82ms  min=1903.33ms med=2270.24ms  max=3287.80ms  p(90)=2686.29ms  p(95)=2849.89ms 
     ✓ { endpoint:diffs }.................: avg=330.37ms   min=161.82ms  med=342.22ms   max=1032.38ms  p(90)=470.05ms   p(95)=518.30ms  
   ✓ http_reqs............................: 736     12.266651/s
     ✓ { endpoint:diffs_batch.json }......: 510     8.499989/s
     ✓ { endpoint:diffs_metadata.json }...: 111     1.849998/s
     ✓ { endpoint:diffs }.................: 115     1.916664/s
     iteration_duration...................: avg=10514.76ms min=0.15ms    med=10506.32ms max=13304.87ms p(90)=11798.42ms p(95)=12374.22ms
     iterations...........................: 95      1.583331/s
   ✓ successful_requests..................: 100.00% ✓ 732  ✗ 0   
     vus..................................: 1       min=1  max=20
     vus_max..............................: 20      min=20 max=20

The results above show that the diffs_batch.json action has a P90 of 2439.88ms and diffs_metadata.json 2849.89ms. These were tested on our 10k Reference Architecture with a RPS target of 20/s against the page (spread across each controller action). It targeted an MR on our own gitlabhq project that has a typically high number of changes - ~678+.

As per our new server performance targets these controller actions are above our main target of 500ms and fall under the ~S2 tier. Task is to improve each action's performance into the next tier ~S3 (<2000ms).