Draft: feat(indexer): add Gitaly proxy archive transport
What does this MR do and why?
The code indexer currently downloads repository archives through Rails, which makes Rails run and stream git archive. This MR adds an opt-in Workhorse Gitaly proxy path with per-call Rails fallback, bounded whole-stream retries, and an archive root compatible with the production extractor. The default remains rails_http.
Related Issues
Related to #1252
Chained on !2381 (targets its branch). Server side: gitlab-org/gitlab!253414, gitlab-org/gitlab!253431, gitlab-org/gitlab!253417.
Testing
cargo test -p indexer --lib (505 passed), cargo test -p gitlab-client (42 passed), and cargo clippy -p gitlab-client -p indexer --all-targets --all-features -- -D warnings.
Performance Analysis
The default Rails path has no performance change. The opt-in proxy path writes each archive to a temporary file before extraction, adding one local disk write and delaying extraction until the download completes.
- This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context: implementation details, E2E finding, retry policy, and design decisions
- The indexer adds a repository service backed by Gitaly's
RepositoryService.GetArchive; project metadata continues to use Rails HTTP. gitlab.gitaly_transportsupportsrails_http,workhorse_ws, andworkhorse_ws_with_fallback. The last mode falls back for unavailable handshakes and proxy policy drift, but not after archive bytes arrive.- E2E against the complete K2, Rails, Workhorse, Praefect, and Gitaly chain found that an empty
GetArchive.prefixproduced slash-rooted entries. The production extractor selected the first top-level file as the archive root and returned no files, while the Rails transport extracted 41 files. - The proxy request now derives the same prefix shape as the Rails archive path from the Rails-vouched
gl_project_pathand requested ref. It stripsrefs/heads/orrefs/tags/and replaces ref path separators with hyphens, matching Rails archive metadata construction without a fixed archive-root literal. - The in-process Workhorse harness records
GetArchiverequests. The indexer test asserts the derivedproj-feature-archiveprefix forgroup/projatrefs/heads/feature/archive. - Rails resolves the requested ref before calling Gitaly, while the proxy can send the ref directly. Archive metadata and bytes can therefore differ between transports; parity is the extracted file set, not byte equality.
- The archive path classifies stale sessions, admission backpressure, stream deadlines, and stream cuts. K1 owns one stale-session retry. K2 applies bounded full-jitter backoff before a stream starts and restarts a cut stream from zero up to three total attempts.
- Transport, restart, and stream-deadline metrics record one outcome per logical archive. A fallback records the proxy attempt once rather than counting the nested Rails fetch separately.
- The fake Workhorse moved into shared GitLab client test support and covers archive streaming, stream cuts, and request capture.
- Archives are buffered in a temporary file instead of streaming directly into the tar reader. This prevents partial bytes from one attempt from reaching a restarted attempt.
- Review clarification added bounded pre-stream backoff for proxy
RESOURCE_EXHAUSTEDadmission responses, includingmax_concurrent_streams. - This MR depends on !2381. Server-side changes are gitlab-org/gitlab!253414, gitlab-org/gitlab!253431, and gitlab-org/gitlab!253417.
- The in-process harness exercises WebSocket and tonic behavior. This MR does not add a separate Docker test. Channel caching, admission controls, cool-down, and webserver blob reads remain follow-up scope.