Propagate caller_id and feature_category from Rails ApplicationContext into Gitaly gRPC metadata

Summary

During incident INC-13016, responders could not tell where elevated Gitaly traffic came from. They could not separate crawlers, background jobs, and real users.

Rails logs have meta.caller_id to identify the caller. For GraphQL the request-level meta.caller_id is GraphqlController#execute. It does not contain the operation name. Example from production logs:

  • meta.caller_id: GraphqlController#execute
  • graphql.operation_name: CommitReferences

So the operation name is logged in a separate field, not in the caller_id that Gitaly would inherit.

Two things block useful Gitaly attribution:

  1. The operation name is not in the caller_id while the query runs.
  2. The caller_id is never forwarded to Gitaly.

Current behavior

caller_id during execution

Gitlab::Graphql::Tracers::InstrumentationTracer#execute_multiplex runs the queries via super. All Gitaly calls happen inside that call. The tracer only sets ApplicationContext.with_context(caller_id: "graphql:<operationName>") afterwards, in the ensure block, and only to write the GraphqlLogger line and the SLI metric. So graphql:<operationName> exists in the GraphQL log, but not during execution.

During execution the active caller_id is the controller endpoint, GraphqlController#execute. The controller does not set a per-operation caller_id. So any Gitaly call made while resolving a GraphQL query sees GraphqlController#execute.

This is why the earlier attempt in !216756 (closed) is still relevant. It tried to add the operation name to the execution-time caller_id. It was closed as redundant, but that rationale was about the GraphqlLogger line, which already has graphql:<op>. It was not about the execution path or the request-level caller_id.

Metadata sent to Gitaly

Gitlab::GitalyClient.application_context_metadata in lib/gitlab/gitaly_client.rb (around line 431) copies these fields from Gitlab::ApplicationContext.current into Gitaly gRPC metadata:

  • username from meta.user
  • user_id from meta.user_id
  • gl_user_id from meta.gl_user_id
  • remote_ip from meta.remote_ip

It does not copy meta.caller_id or meta.feature_category.

Note that the existing call_site metadata field is the Gitaly RPC feature name. It is not the Rails caller_id. These are different things and should not be confused.

Proposal

Both parts are needed. Forwarding caller_id alone would attribute every GraphQL Gitaly call to GraphqlController#execute, which is the same useless value seen during the incident.

  1. Include the GraphQL operation name in the caller_id during query execution, so it is the active caller_id when Gitaly calls are made. This revives the intent of merge request 216756.
  2. Add caller_id (from meta.caller_id) and feature_category (from meta.feature_category) to the metadata built in application_context_metadata.
  3. Update Gitaly so it reads and logs both fields in its structured logs, next to the existing fields.

Acceptance criteria

  • A GraphQL-originated Gitaly call produces a populated caller_id field in Gitaly structured logs that includes the operation name, for example graphql:CommitReferences.
  • Both authenticated and unauthenticated GraphQL request paths show the correct caller_id.
  • The change is validated in canary (cny) before it rolls out more broadly.

Related

Edited by Stan Hu