Branches List API performance has degraded notably

On the latest Omnibus Nightly packages the Branches List API has degraded notably in performance over the last week:

* Environment:                10k
* Environment Version:        13.12.0-pre `4366b711516`
* Option:                     60s_200rps
* Date:                       2021-05-05
* Run Time:                   1h 30m 33.87s (Start: 11:54:45 UTC, End: 13:25:19 UTC)
* GPT Version:                v2.8.0

NAME                                                     | RPS   | RPS RESULT           | TTFB AVG  | TTFB P90              | REQ STATUS     | RESULT 
---------------------------------------------------------|-------|----------------------|-----------|-----------------------|----------------|--------
api_v4_projects_repository_branches                      | 200/s | 127.16/s (>160.00/s) | 1445.81ms | 2183.27ms (<500ms)    | 100.00% (>99%) | FAILED²

The endpoint previously was previously polling around the 100-200ms mark.

Screenshot_2021-05-06_at_10.51.20

Looking at the metrics Gitaly appears to be getting hit hard now by this endpoint at scale:

Screenshot_2021-05-06_at_15.49.13

Edited by Grant Young