Optimize remote CI artifact parsing and size validation (behind FF)
What does this MR do and why?
Makes parsing of remote (object storage) CI artifacts in Ci::Artifactable#each_blob more efficient, behind the new ci_optimize_artifact_parsing feature flag (gitlab_com_derisk, disabled by default, project actor):
- Download the file once instead of twice. Currently
Gitlab::Ci::Artifacts::DecompressedArtifactSizeValidator#valid_on_storage?downloads the full file into a tempfile to validate its decompressed size and discards it, thenfile.opendownloads the file again throughGitlab::HttpIO— which issues a separate HTTP request on a fresh connection per 128 KB chunk — for the actual parsing. With the flag enabled, the file is downloaded once using the existingObjectStorage::Concern#use_open_filehelper, and the local copy is reused for both the decompressed-size validation and the parsing. This halves the number of downloads and replaces the per-chunk ranged reads with a single streaming GET. - Stop decompressing once the size limit is exceeded.
Gitlab::Ci::DecompressedGzipSizeValidatorpipesgzip -dcthroughwc -c, so an archive far over the limit is still decompressed in full before the count is compared. With the flag enabled,head -c (max_bytes + 1)is inserted into the pipeline: the size check result is preserved (a capped count still exceeds the limit), butheadcloses the pipe at the cap sogzipterminates shortly after the limit instead of decompressing the entire archive.
Validation outcomes are unchanged: the same size validator runs against the same bytes, and a file that exceeds the limit still raises DecompressedArtifactSizeValidator::FileDecompressionError. Local-storage artifacts keep the current download behavior.
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/605350
- Rollout issue: #605502
How to set up and validate locally
-
Configure object storage for artifacts in GDK and set
direct_upload: true. -
Run a pipeline with a job that produces a JUnit report artifact (
artifacts:reports:junit). -
In a Rails console, enable the flag and parse the artifact:
Feature.enable(:ci_optimize_artifact_parsing, project) artifact = project.job_artifacts.where(file_type: 'junit').last artifact.each_blob { |blob, name| puts blob.bytesize } -
Observe (for example via mitmproxy or object storage access logs) that the file is fetched once, instead of once in full plus once per 128 KB chunk.