Skip to content

Support for logging propagated client identity

Igor requested to merge client-identity into master

It has frequently been the case that we are seeing an increase in traffic to a certain gitaly node, but then have no easy way of figuring out where that traffic came from.

We often do have corresponding logs in other "front-door" services, such as workhorse and gitlab-shell -- but correlating with those logs is often not so easy.

  • It requires knowing which front door is causing the traffic.
  • It then requires potentially translating filters to the logs for the other dataset, on fields which may not be present.
  • While correlation_id allows joining individual requests, what we often care about during an incident is the analysis in aggregate -- Elasticsearch does not support that kind of joining.

What would often help operators with this kind of diagnosis is the ability to look at the origin of the traffic directly from within the gitaly logs.

The main fields that would help are:

  • remote_ip: For anonymous traffic this is super useful. We do not currently have this for SSH requests (https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10954), but that's somewhat okay because we always have the username in that case.
  • username, user_id: While a huge chunk of the traffic to gitaly is anonymous, it's still useful to see the user identity in the case where it's not. And to be able to do so without having to join across datasets.

This patch adds the capability to gitaly. Clients need to be update to actually propagate this value. Here is a proposed patch for gitlab-shell: https://gitlab.com/gitlab-org/gitlab-shell/-/compare/master...client-identity.

refs gitlab#232645 (closed)

Merge request reports