Pass feature_category information to gitaly

Recently, feature categories are captured inside Rails Web, Rails API, and Rails Sidekiq workers. That information is used heavily in the Stage Group Dashboards. Downstream services, only Gitaly at the moment AFAIK, can obtain that information and append to their Prometheus metrics, or Jaeger tracing. If feature category information is available in those platforms, we can aggregate the captured data and append it to the stage group dashboards. It would be a great addition to the observability of the production, while the effort is reasonable.

Proposal

Thankfully we are using gRPC to interact with Gitaly, it supports interceptor (middleware) at both client-side and server-side. The feature category information can be packed into gRPC's call metadata (just like labkit-ruby's correlation). Then it is unpacked at the Gitaly's server interceptors.

• Create a client Interceptor, inject into https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/gitaly_client.rb#L48
• Create a server interceptor to unwrap the feature category from the client, put it into the request context
• Create a new server interceptor to replace go-grpc-prometheus interceptor as the default one does not support custom labels.
• Verify to see whether Labkit tracing is able to capture the feature category.
• Revisit the runbook aggregation to add feature_category

Risks

• Gitaly metrics cardinality