## Summary With `FF_USE_PARALLEL_CACHE_TRANSFER` or `FF_USE_PARALLEL_ARTIFACT_TRANSFER` enabled, every chunk is fetched with its own Range GET, and none of them says which version of the object it expects. If the object is replaced while the chunks are in flight, for example when another job uploads the same cache key, the chunks requested after the replacement come from the new version. Each answer is a correct `206` for its range, so the download succeeds with a file that matches neither version. On the presigned cache path the single-byte probe reads the object's `ETag`, but `downloadParallel` only logs it ([`cache_extractor.go#L306`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/b130a28af451b8b1044ad68b7c77e33d1669eb63/commands/helpers/cache_extractor.go#L306), [`#L527-L528`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/b130a28af451b8b1044ad68b7c77e33d1669eb63/commands/helpers/cache_extractor.go#L527-L528)), and the artifact probe does not read it. The chunk requests of both carry only `Range` ([`cache_extractor.go#L339-L345`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/b130a28af451b8b1044ad68b7c77e33d1669eb63/commands/helpers/cache_extractor.go#L339-L345), [`gitlab.go#L1218-L1219`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/b130a28af451b8b1044ad68b7c77e33d1669eb63/network/gitlab.go#L1218-L1219)). The sequential download makes one GET and cannot mix versions. It was found in the review of !7635 ([note](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/7635#note_4010945248)), with a run against SeaweedFS 3.85 and LocalStack 4.4.0 in which the object was replaced after the first chunk response: the download returned no error and the archive matched neither version. The same review checked that a presigned GET on both stores accepts an unsigned `If-Match` header and answers `206` when it matches and `412` when it is stale. ## Steps to reproduce On `main` (b130a28a), add this test to `commands/helpers/cache_extractor_test.go` (it uses that file's imports and helpers) and run `go test -run TestPresignedParallelObjectReplaced -v ./commands/helpers/`. The `httptest` server answers the first half of the object from the old version and the rest from the new one, as a store does when the object is replaced between those requests. ```go func TestPresignedParallelObjectReplaced(t *testing.T) { cdTempDir(t) v1 := bytes.Repeat([]byte{'1'}, 64) v2 := bytes.Repeat([]byte{'2'}, 64) lm := time.Date(2020, 5, 1, 12, 0, 0, 0, time.UTC).Format(http.TimeFormat) ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { start, end, ok := parseBytesRangeHeader(r.Header.Get("Range")) if !ok { http.Error(w, "Range expected", http.StatusBadRequest) return } obj, etag := v1, `"v1"` if start >= 32 { obj, etag = v2, `"v2"` } w.Header().Set("Last-Modified", lm) w.Header().Set("ETag", etag) w.Header().Set("Content-Range", fmt.Sprintf("bytes %d-%d/%d", start, end, len(obj))) w.WriteHeader(http.StatusPartialContent) _, _ = w.Write(obj[start : end+1]) })) t.Cleanup(ts.Close) cmd := CacheExtractorCommand{File: cacheExtractorArchive, URL: ts.URL + "/cache.zip", ChunkSize: 8, Concurrency: 4} done, err := cmd.tryPresignedParallelDownload() got, _ := os.ReadFile(cacheExtractorArchive) t.Logf("done=%v err=%v archive=%q", done, err, got) } ``` It prints: ```plaintext done=true err=<nil> archive="1111111111111111111111111111111122222222222222222222222222222222" ``` ## Actual behavior The download succeeds and leaves a file made of two versions of the object. The cache archive keeps the probe's `Last-Modified`, so a later run takes it as up to date. ## Expected behavior A chunk of another version than the one the probe saw fails the parallel download, and the download falls back to a single GET. ## Relevant logs and/or screenshots The test output above. ## Environment description Any executor, with `FF_USE_PARALLEL_CACHE_TRANSFER` (presigned cache URLs) or `FF_USE_PARALLEL_ARTIFACT_TRANSFER` (artifacts with `direct_download` redirected to object storage) enabled, and an object replaced while it is being downloaded. Both flags default to `false`. ### Used GitLab Runner version GitLab Runner v19.5.0, whose `cache_extractor.go` and `network/gitlab.go` are the same as at the commit below. `main` at b130a28af451b8b1044ad68b7c77e33d1669eb63, and the same code since both parallel downloads were added in v19.0.0. ## Possible fixes Send `If-Match` with the probe's `ETag` on every chunk request, and treat a `412` like a chunk that does not hold the requested range: fail the parallel download and fall back to the single GET. A weak `ETag` never matches `If-Match`, so the probe's `ETag` has to be strong for this to apply; without one, the download can keep today's behavior.