Improve performance of show action for Projects::MergeRequestsController into main tier
Following a review done in conjunction with the groupcode review team a new target MR was selected for performance testing that was more realistic and representative of the 99th percentile of all MRs on .com. In addition to this tweaks were made to our tests to be up to date and more focused on the MR pages endpoints.
Testing against the Merge Request Changes page for the selected MR at scale found that the Projects::MergeRequestsController#show
controller and action specifically was unperformant:
* Environment: 10k
* Environment Version: 15.2.0-pre `bae4f991b34`
* Option: 60s_200rps
* Date: 2022-07-07
* Run Time: 1h 27m 41.13s (Start: 04:46:50 UTC, End: 06:14:31 UTC)
* GPT Version: v2.10.0
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|-------|----------------------|-----------|----------------------|----------------|--------
web_project_merge_request | 20/s | 19.66/s (>16.00/s) | 292.16ms | 508.99ms (<1800ms) | 100.00% (>99%) | Passed¹
█ Web - Project Merge Request Page
data_received....................................................................: 189 MB 3.0 MB/s
data_sent........................................................................: 178 kB 2.8 kB/s
group_duration...................................................................: avg=4633.78ms min=659.92ms med=4966.78ms max=5550.84ms p(90)=5103.57ms p(95)=5146.24ms
http_req_blocked.................................................................: avg=0.48ms min=0.00ms med=0.71ms max=18.08ms p(90)=0.93ms p(95)=0.97ms
http_req_connecting..............................................................: avg=0.41ms min=0.00ms med=0.62ms max=1.12ms p(90)=0.82ms p(95)=0.87ms
http_req_duration................................................................: avg=294.48ms min=62.61ms med=118.78ms max=3641.53ms p(90)=514.19ms p(95)=646.20ms
{ expected_response:true }.....................................................: avg=294.48ms min=62.61ms med=118.78ms max=3641.53ms p(90)=514.19ms p(95)=646.20ms
http_req_failed..................................................................: 0.00% ✓ 0 ✗ 1236
http_req_receiving...............................................................: avg=2.25ms min=0.05ms med=0.12ms max=13.10ms p(90)=8.80ms p(95)=9.21ms
http_req_sending.................................................................: avg=0.07ms min=0.02ms med=0.06ms max=0.58ms p(90)=0.10ms p(95)=0.11ms
http_req_tls_handshaking.........................................................: avg=0.00ms min=0.00ms med=0.00ms max=0.00ms p(90)=0.00ms p(95)=0.00ms
http_req_waiting.................................................................: avg=292.16ms min=62.40ms med=118.58ms max=3635.17ms p(90)=508.99ms p(95)=643.79ms
✓ { controller:Projects::MergeRequests::ContentController#cached_widget.json }...: avg=121.60ms min=99.94ms med=114.60ms max=563.01ms p(90)=141.06ms p(95)=152.30ms
✓ { controller:Projects::MergeRequests::ContentController#widget.json }..........: avg=110.50ms min=94.03ms med=106.86ms max=232.43ms p(90)=123.25ms p(95)=136.93ms
✓ { controller:Projects::MergeRequestsController#discussions.json }..............: avg=628.19ms min=407.54ms med=469.31ms max=3635.17ms p(90)=524.11ms p(95)=2071.42ms
✓ { controller:Projects::MergeRequestsController#show.json }.....................: avg=75.45ms min=62.40ms med=71.49ms max=123.42ms p(90)=91.18ms p(95)=99.38ms
✓ { controller:Projects::MergeRequestsController#show }..........................: avg=524.11ms min=421.94ms med=478.66ms max=790.79ms p(90)=669.74ms p(95)=694.78ms
✓ http_reqs........................................................................: 1236 19.664902/s
✓ { controller:Projects::MergeRequests::ContentController#cached_widget.json }...: 247 3.929798/s
✓ { controller:Projects::MergeRequests::ContentController#widget.json }..........: 247 3.929798/s
✓ { controller:Projects::MergeRequestsController#discussions.json }..............: 247 3.929798/s
✓ { controller:Projects::MergeRequestsController#show.json }.....................: 247 3.929798/s
✓ { controller:Projects::MergeRequestsController#show }..........................: 247 3.929798/s
iteration_duration...............................................................: avg=4617.34ms min=549.86ms med=4965.88ms max=5550.87ms p(90)=5103.35ms p(95)=5146.13ms
iterations.......................................................................: 247 3.929798/s
✓ successful_requests..............................................................: 100.00% ✓ 1235 ✗ 0
vus..............................................................................: 4 min=2 max=20
vus_max..........................................................................: 20 min=20 max=20
This was an interesting find as the show
action amounts to the main page load more than anything else, which isn't typically a performance failure these days.
The endpoint was tested on our 10k Reference Architecture with a RPS target of 20/s against the page (spread across each controller action).
As per our performance targets the controller action is above our main target of 200ms and fall under the severity4 tier. Task is to improve the action's performance into the main tier (<200ms).