Improve performance of Repository Compare Branches/Tags API under load into next tier

After adding in a new performance test for the Branches Repository Compare API it was found that endpoint is unperformant:

 █ Results summary
* Environment:                10k
* Environment Version:        13.11.0-pre `8b407f0a3fb`
* Option:                     60s_200rps
* Date:                       2021-04-20
* Run Time:                   1m 4.11s (Start: 16:49:59 UTC, End: 16:51:03 UTC)
* GPT Version:                v2.7.0
NAME                                        | RPS   | RPS RESULT        | TTFB AVG  | TTFB P90            | REQ STATUS     | RESULT         
--------------------------------------------|-------|-------------------|-----------|---------------------|----------------|----------------
api_v4_projects_repository_compare          | 200/s | 68.5/s (>48.00/s) | 2563.20ms | 3392.07ms (<5000ms) | 100.00% (>99%) | Passed

Looking at the environment metrics it appears that the endpoint is hitting the Gitaly and Praefect massively: api_v4_projects_repository_compare_branches_tags

The testing was done on our test 10k Reference Architecture environment as standard at 200 RPS with the project being tested a copy of gitlabhq (tarball can be found here). For the repository comparison these branches are being used - https://staging.gitlab.com/gpt/large_projects/gitlabhq1/-/compare/12-2-stable...12-1-stable - diff size is 200 commits, 546 changed files with 8342 additions and 2856 deletions.

Steps to reproduce

  1. Check out the Performance Tool
  2. Run the specific test with the run-k6 command.

What is the current bug behavior?

The results above show that the Branches/Tags Repository Compare API has a P90 of 5517.95ms.

What is the expected correct behavior?

As per our performance targets this endpoint is above the second tier target of 2000ms which is severity2 . Task is to improve the endpoint's performance into next tier.

Edited by Nailia Iskhakova