Send gitaly traces to stackdriver trace
We're still blocked on Jaeger &210 (closed) due to competing priorities.
We're getting alerted quite a lot on gitaly at the moment: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13571. It's quite challenging to tell where the time is being spent in a slow RPC. Distributed tracing would really help with diagnosis here.
In order to accelerate the efforts and get insights more quickly, I'm proposing we integrate gitaly with Stackdriver Trace (GCP's hosted tracing product). It's not as fancy as jaeger, and I still think we will want to go to jaeger eventually. But this should help close the o11y gap we have there.
Rough steps of this work:
-
Add Stackdriver Trace support to labkit gitlab-org/labkit!116 (merged) -
Bump labkit dependency on gitaly gitlab-org/gitaly!3637 (merged) -
Set up service account permissions -
Reconfigure gitaly on gstg https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/337 -
Reconfigure gitaly on gprd https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/342 -
Tune sampling ratio if needed https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/346
Status
This has been rolled out to the production gitaly fleet with a sampling ratio of 0.01%.
cc @andrewn