Draft: feat(indexer): add Gitaly proxy archive transport

What does this MR do and why?

The code indexer currently downloads repository archives through Rails, which makes Rails run and stream git archive. This MR adds an opt-in Workhorse Gitaly proxy path with per-call Rails fallback, bounded whole-stream retries, and an archive root compatible with the production extractor. The default remains rails_http.

Related to #1252

Chained on !2381 (targets its branch). Server side: gitlab-org/gitlab!253414, gitlab-org/gitlab!253431, gitlab-org/gitlab!253417.

Testing

cargo test -p indexer --lib (505 passed), cargo test -p gitlab-client (42 passed), and cargo clippy -p gitlab-client -p indexer --all-targets --all-features -- -D warnings.

Performance Analysis

The default Rails path has no performance change. The opt-in proxy path writes each archive to a temporary file before extraction, adding one local disk write and delaying extraction until the download completes.

  • This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context: implementation details, E2E finding, retry policy, and design decisions
  • The indexer adds a repository service backed by Gitaly's RepositoryService.GetArchive; project metadata continues to use Rails HTTP.
  • gitlab.gitaly_transport supports rails_http, workhorse_ws, and workhorse_ws_with_fallback. The last mode falls back for unavailable handshakes and proxy policy drift, but not after archive bytes arrive.
  • E2E against the complete K2, Rails, Workhorse, Praefect, and Gitaly chain found that an empty GetArchive.prefix produced slash-rooted entries. The production extractor selected the first top-level file as the archive root and returned no files, while the Rails transport extracted 41 files.
  • The proxy request now derives the same prefix shape as the Rails archive path from the Rails-vouched gl_project_path and requested ref. It strips refs/heads/ or refs/tags/ and replaces ref path separators with hyphens, matching Rails archive metadata construction without a fixed archive-root literal.
  • The in-process Workhorse harness records GetArchive requests. The indexer test asserts the derived proj-feature-archive prefix for group/proj at refs/heads/feature/archive.
  • Rails resolves the requested ref before calling Gitaly, while the proxy can send the ref directly. Archive metadata and bytes can therefore differ between transports; parity is the extracted file set, not byte equality.
  • The archive path classifies stale sessions, admission backpressure, stream deadlines, and stream cuts. K1 owns one stale-session retry. K2 applies bounded full-jitter backoff before a stream starts and restarts a cut stream from zero up to three total attempts.
  • Transport, restart, and stream-deadline metrics record one outcome per logical archive. A fallback records the proxy attempt once rather than counting the nested Rails fetch separately.
  • The fake Workhorse moved into shared GitLab client test support and covers archive streaming, stream cuts, and request capture.
  • Archives are buffered in a temporary file instead of streaming directly into the tar reader. This prevents partial bytes from one attempt from reaching a restarted attempt.
  • Review clarification added bounded pre-stream backoff for proxy RESOURCE_EXHAUSTED admission responses, including max_concurrent_streams.
  • This MR depends on !2381. Server-side changes are gitlab-org/gitlab!253414, gitlab-org/gitlab!253431, and gitlab-org/gitlab!253417.
  • The in-process harness exercises WebSocket and tonic behavior. This MR does not add a separate Docker test. Channel caching, admission controls, cool-down, and webserver blob reads remain follow-up scope.
Edited by Dmitry Gruzd

Merge request reports

Loading
Loading