Attribute CI job git-over-HTTP traffic to Gitaly

What does this MR do and why?

Gitaly cannot tell a CI job fetching a repository from a developer doing the same clone. Both arrive with client_name=gitlab-workhorse, so CI traffic (a large share of git fetch volume) is invisible in Gitaly telemetry as a distinct source.

This MR sets call_metadata['client_name'] = 'gitlab-ci-job' (constant CI_JOB_CLIENT_NAME) in Gitlab::Workhorse.git_http_ok, at the point where job-token auth is already detected to set GlBuildID. No Workhorse or Gitaly change is needed, and no new label is added.

Why this happens and why the fix is safe

Two structural reasons CI is invisible today:

  • client_name names the calling binary. CI is deliberately an ordinary git client over HTTPS, on the same code path as a human, so it looks identical at that layer.
  • CI clones never go through Gitlab::GitalyClient, so the Rails-emitted per-caller metric gitlab_sli_gitaly_client_calls_total (type/stage/shard labels) structurally cannot see them.

Gitlab::Workhorse.git_http_ok already knows whether the request is job-token authenticated, since it uses that to set GlBuildID. This MR reuses that same signal to set client_name in call_metadata.

No downstream change is required because Workhorse already allowlists client_name in allowedMetadataKeys and forwards it on Labkit's canonical x-gitlab-client-name gRPC metadata key (workhorse/internal/gitaly/gitaly.go). Gitaly's request-info middleware reads correlation.ExtractClientNameFromContext first and only falls back to the raw per-connection key, so the Rails-supplied value wins over the gitlab-workhorse default. There is existing precedent for this exact mechanism: gkg-indexer (gitaly_server_hash(client_name:)).

The value is gitlab-ci-job, not gitlab-ci-runner: the job token covers every git call a job makes (submodules, go get, scripted fetches), not only the runner's checkout.

Cardinality: this deliberately reuses the existing client_name label rather than adding a new one. client_name gains one more value; there is no new label dimension on a counter that already carries roughly 46,000 series in gprd.

Limiter semantics do not change. Gitaly's concurrency limiter is keyed by repository; client_name only matters to it through internal_clients, and only when the call carries no username. A job-token call carries the job's user (Gitlab::Auth.build_access_token_check returns build.user; git_http_ok forwards it as GL_USERNAME), so these calls stay on the authenticated per-repository budget exactly as before.

Why not the call_site label, which looks like the obvious home for this: it is not in Workhorse's metadata allowlist, and it has been dead since 2018. GitalyClient.migrate was the only writer of the thread-local that request_kwargs reads (Thread.current[:gitaly_feature_stack]), and that writer was removed in commit 35239a6a (2018-11-16) while the reader stayed. Verified empirically: call_site is unknown for 100% of series in gprd (45,952), gstg, pre, and ops. Not fixed here; flagged as a separate cleanup worth doing.

References

  • Rollout issue: #631984
  • Dashboards that filter on client_name and need a CI flow before the flag is enabled: gitlab-com/runbooks!11787 (merged) (dashboards/gitaly/ingress-flow.dashboard.jsonnet uses byClient('gitlab-workhorse') for the git-over-HTTPS flow; traffic-sources.dashboard.jsonnet describes CI as not separable). Nothing in metrics-catalog/ or mimir-rules/ filters on client_name.
  • Follow-up for attributing CI traffic to a job rather than a source (needs Gitaly to log a new metadata key, then Workhorse to forward it): gitaly#7473.

It came out of an investigation into Gitaly traffic sources on the canary node, where CI traffic could not be separated from human clones.

Screenshots or screen recordings

Not applicable. No UI change.

How to set up and validate locally

  1. Enable the flag in a rails console: Feature.enable(:ci_job_token_gitaly_client_name)
  2. Run a CI job that clones a project over HTTPS.
  3. Confirm Gitaly attributes the RPC to gitlab-ci-job instead of gitlab-workhorse. This is visible either in Gitaly's logs via the grpc.meta.client_name field, or on the gitaly_service_client_requests_total metric's client_name label.

Feature flag: ci_job_token_gitaly_client_name, type gitlab_com_derisk, default_enabled: false. It is flagged because it shifts the label distribution of a production metric; the flag is removed once it has been on fleet-wide and the dashboards above are merged (2-month lifespan per the gitlab_com_derisk rules). The flag name deliberately sits outside the gitaly_ prefix: Feature::Gitaly.server_feature_flags forwards every flag with that prefix to Gitaly as gitaly-feature-* metadata.

MR acceptance checklist

  • Tests added: spec/lib/gitlab/workhorse_spec.rb covers flag on (the spec default), flag off, and the non-CI request case, and asserts GlBuildID is preserved in every branch.
  • Changelog: not needed. This is internal observability metadata only, flag-gated and off by default, with no user-facing behavior change.
  • RuboCop and RSpec were not run locally (no bundle install in this environment); CI is the lint and test run for this change.
Edited by Kranthi Erusu

Merge request reports

Loading
Loading