FindCommit for a ref slows down dramatically with many concurrent requests

While investigating some FindCommit slowdowns on the gitlab-org/gitlab project in gitlab-com/gl-infra/scalability#661, I noticed that they were correlated with many concurrent requests to the raw endpoint.

We're going to remove the FindCommit calls on the raw endpoint in gitlab-com/gl-infra/scalability#677, but we should still investigate this. For instance, using hey I generated a bunch of concurrent requests on staging to another endpoint that uses FindCommit:

$ hey -n 2000 -c 100 https://staging.gitlab.com/gitlab-org/gitlab/-/commits/master?format=json

Summary:
  Total:	57.0079 secs
  Slowest:	9.0723 secs
  Fastest:	0.6443 secs
  Average:	2.4484 secs
  Requests/sec:	35.0828


Response time histogram:
  0.644 [1]	|
  1.487 [908]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  2.330 [326]	|■■■■■■■■■■■■■■
  3.173 [156]	|■■■■■■■
  4.016 [149]	|■■■■■■■
  4.858 [175]	|■■■■■■■■
  5.701 [149]	|■■■■■■■
  6.544 [91]	|■■■■
  7.387 [29]	|■
  8.229 [9]	|
  9.072 [7]	|


Latency distribution:
  10% in 0.8237 secs
  25% in 0.9540 secs
  50% in 1.7217 secs
  75% in 3.7936 secs
  90% in 5.2561 secs
  95% in 6.0092 secs
  99% in 7.3098 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0042 secs, 0.6443 secs, 9.0723 secs
  DNS-lookup:	0.0002 secs, 0.0000 secs, 0.0058 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0017 secs
  resp wait:	2.4414 secs, 0.6438 secs, 9.0719 secs
  resp read:	0.0027 secs, 0.0003 secs, 0.0279 secs

Status code distribution:
  [200]	2000 responses

In the logs I can see FindCommit get much slower during that period:

I also tried an endpoint (https://staging.gitlab.com/gitlab-org/gitlab/-/commit/4c17044a8039eb51b4c9eb6441a451e8fb4c50cf.diff) that would use FindCommit with a SHA, not a ref, and that did not have the same effect.