Read artifact archives through ranged requests in get_artifact_file
What does this MR do and why?
Implements the ranged-read option discussed in #629157 (closed) for the get_artifact_file MCP tool. The approach was proposed by Vitali in !254894 (comment 3835046960)
What changed
- The tool no longer streams the whole artifacts archive into a Tempfile on the Puma node (the
use_open_fileapproach). It now callsZip::File.open_bufferdirectly on the IO object the uploader returns. - That IO is
Gitlab::HttpIOwhen the archive lives in remote object storage (fetched with 128 KBRangerequests), or a plainFilewhen it is on local storage. - New class
Mcp::Tools::RangedZipIo: a delegator that makesHttpIOacceptable to the rubyzip library. It does two things:- Adds an
eofprobe method, becauseHttpIOonly exposeseof?. - Hides
HttpIO#path. Without this,Zip::Filepicks up the path and uses it as the archive filename, and reopening entries later fails with ENOENT.
- Adds an
- The adapter's code comment documents that it supports sequential reads only. This is a known, intentional limitation.
Spike measurement
Using an in-process HTTP server that honors Range requests, reading a single entry out of a 922 KB test archive fetched 267 KB across 3 requests: two small tail chunks for the central-directory scan, plus one chunk at the target entry's offset. The request count does not grow with archive size — it stays at 3 regardless of how large the archive is, because it is driven by the zip format's structure (central directory at the end, then a seek to the entry), not by the total bytes.
Test coverage
- New spec using a large archive in object storage, asserting the ranged fetch stays under half the size of the full archive.
- Unit tests for
Mcp::Tools::RangedZipIo. - Examples for a failed ranged request and a non-zip archive body, both asserting the download-URL error.
- All existing behavior specs for this tool pass unchanged.
Limits
- The 20 MB
MAX_ARCHIVE_BYTEScap stays. Ranged reads remove the transfer and disk cost, butZip::File.open_bufferstill loads the whole central directory and builds oneZip::Entryper record, and theMAX_ARCHIVE_ENTRIEScheck only runs when the metadata file exists. The archive size cap is what bounds that load for an archive without metadata. It can be raised once production numbers are in. MAX_BYTE_OFFSETis tied toMAX_ARCHIVE_BYTES. WithHttpIO, skipping into a stored (uncompressed) entry costs one 128 KBRangerequest per chunk, so the offset ceiling also bounds the request count per call. A stored entry can never be larger than its archive, so the archive cap is the natural ceiling for the offset too. This settles open question 3 in #629157 (closed).- A
Gitlab::HttpIO::FailedToGetChunkErrororZip::Errorduring the read returns an error that names the job and its download URL, instead of a generic tool failure.
The companion application setting proposed in !255712 (closed) is not needed with these limits and that MR is closed.
Related to #629157 (closed)