Repeating alerts for Gitaly latency in cny

We're seeing repeating alerts for Gitaly apdex on file-cny-01 in production. Could we please get some help troubleshooting these?

see: gitlab-com/gl-infra/production#4142 (closed) for the latest incident (and some initial troubleshooting notes)

Last 6h:

src: https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly-host-detail?viewPanel=4080926127&orgId=1&from=now-6h%2Fm&to=now%2Fm&var-PROMETHEUS_DS=Global&var-environment=gprd&var-fqdn=file-cny-01-stor-gprd.c.gitlab-production.internal

It's hard to determine when these alerts started. We had similarly looking incidents months ago, but silences were created in the meantime, other incidents were taking palce and multiple changes were applied in production. Here's last 7d:

src: https://thanos.gitlab.net/graph?g0.range_input=7d&g0.max_source_resolution=0s&g0.expr=min_over_time(gitlab_service_node_apdex%3Aratio_5m%7Benv%3D%22gprd%22%2Cenvironment%3D%22gprd%22%2Cmonitor%3D%22global%22%2Ctype%3D%22gitaly%22%7D%5B1m%5D)%0A&g0.tab=0

Not sure at this point if this is related or not, but we're also getting occasional alerts for apdex on other nodes:

src: https://thanos.gitlab.net/graph?g0.range_input=6h&g0.max_source_resolution=0s&g0.expr=min_over_time(gitlab_service_node_apdex%3Aratio_5m%7Benv%3D%22gprd%22%2Cenvironment%3D%22gprd%22%2Cmonitor%3D%22global%22%2Ctype%3D%22gitaly%22%7D%5B1m%5D)%0A&g0.tab=0