`gitlab-org/gitlab` HTTP clones are waiting up to 10 minutes to acquire a concurrency lock.

Spun out of scalability#64 (comment 254346588)

gitlab-org/gitlab HTTP clones are waiting up to 10 minutes to acquire a concurrency lock.

p95 concurrency lock acquisition for gitlab-org/gitlab

https://log.gitlab.net/goto/9dcf3572bca2fedb8d77b5430708cb4a

I'm fairly certain that this would also explain this alert, which we've been seeing a lot of of late

A few other things that correlate these events:

1. The event is firing for the cny stage, not main. Since this is where gitlab-org/gitlab traffic is routed, it seems to be correlated.
2. Git HTTP active connections over 24 hours looks very similar to the p95 graph above

https://dashboards.gitlab.net/d/jIYYw9-ik/gitlab-workhorse-alerting?orgId=1&from=now-24h&to=now&panelId=2&tz=UTC&var-environment=gprd&var-type=git&var-stage=cny

This requires a Service::Git scoped label, will add later...

cc @jacobvosmaer-gitlab @zj-gitlab @johncai @pokstad1