Use X-Request-Id for correlation ID from trusted /internal API requests
Right now it's not possible to trace a request all the way through if Gitaly has to make internal API calls back to GitLab Rails. For example, with the new LFS smudge filter added to support #15079 (closed), we have this sequence of events:
sequenceDiagram
autonumber
Client->>+Workhorse: GET /group/project/-/archive/master.zip
Workhorse->>+Rails: GET /group/project/-/archive/master.zip
Rails->>+Workhorse: Gitlab-Workhorse-Send-Data git-archive
Workhorse->>Gitaly: SendArchiveRequest
Gitaly->>Git: git archive master
Git->>Smudge: OID 12345
Smudge->>+Workhorse: GET /internal/api/v4/lfs?oid=12345&gl_repository=project-1234
Workhorse->>+Rails: GET /internal/api/v4/lfs?oid=12345&gl_repository=project-1234
Rails->>+Workhorse: Gitlab-Workhorse-Send-Data send-url
Workhorse->>Smudge: <LFS data>
Smudge->>Git: <LFS data>
Git->>Gitaly: <streamed data>
Gitaly->>Workhorse: <streamed data>
Workhorse->>Client: master.zip
At step 7, the LFS smudge filter makes an API call back to Rails. Even if it attempts to preserve the same correlation ID through the request, at step 8 Workhorse will generate a new correlation ID and use that. As a result, the logs for the /api/v4/internal
calls all have unique correlation IDs when they should share the same one.
The same issue happens when a Gitaly hook has to make an API call to check /api/v4/internal/allowed
, but it's perhaps not as pronounced because there's only 1 or 2 calls.
Some ideas to fix this:
- Add a trusted IP or hostname block that allows Workhorse to use
X-Request-Id
if it is available. - Send a signed token that Workhorse can decode, and if it checks out use the provided correlation ID.
This probably ties into LabKit. What do you think, @andrewn @reprazent?