Refuse a partial answer to the sequential presigned cache download

What does this MR do?

Makes the sequential download of a presigned cache URL fail when its GET, which carries no Range, is answered with anything but 200 OK, before anything is written. A 206 Partial Content was saved as the archive until now. The error points at the likely cause, a caching proxy between the runner and the cache server that replays a stored partial response. An archive already on disk that the answer finds up to date is still used, as before, since nothing is downloaded for it.

Why was this MR needed?

downloadPresignedSequential makes one GET without Range (cache_extractor.go#L322-L342) through getCache, which returns the response with retryOnServerError, and that returns no error for every 2xx (cache_extractor.go#L195-L206, retry_helper.go#L88-L91). A 206 holds only part of the object, but the extractor wrote it as the archive with the object's Last-Modified, so every later run found the archive up to date and kept it, until the object changed.

A caching proxy whose cache key leaves out Range answers a GET without Range that way once it has stored a 206 for an earlier Range request of the same object, such as the bytes=0-0 probe of a runner with FF_USE_PARALLEL_CACHE_TRANSFER enabled. With the caching proxy configuration used in the review of !7635 (merged) (note), nginx 1.31.6 answers a GET without Range, after a bytes=0-0 request, with the stored 206 Partial Content, Content-Range: bytes 0-0/100000 and one byte. The proxy is misconfigured, but the runner could not tell its answer from a complete download. This is also why !7635 (merged) makes the parallel cache download fall back to the sequential GET only for a chunk answered with 200.

I check the status in downloadPresignedSequential, after the up-to-date check and before the body is written. getCache and retryOnServerError stay as they are, because retryOnServerError also judges cache uploads, where other 2xx answers are correct. The error is not retryable, like every answer that retryOnServerError does not count as a server error: a proxy that stored a partial answer gives the same one on the next attempt.

What's the best way to test this MR?

• TestCacheExtractorRemoteServerRejectsPartialContent in commands/helpers: an httptest server answers every request with the one-byte 206 such a proxy replays. With no archive on disk, handlePresignedURL fails with an error naming 206 Partial Content, makes one GET and writes no archive. With an archive on disk that is newer than the answer's Last-Modified, it succeeds and leaves that archive as it was.

go test -race -count=1 -run 'TestCacheExtractorRemoteServer' ./commands/helpers/

With main, the first case fails: the download succeeds and leaves a one-byte archive. The second case passes on main by design: it guards that an up-to-date archive is still used, and checking the status before the up-to-date check fails it.

I ran go test -race -count=1 ./commands/helpers/ with TZ=UTC (in another time zone TestGenerateMetadataToFile, which this MR does not touch, fails; !7665 (merged) fixes it), GOOS=linux, GOOS=windows and GOOS=darwin go vet on it, and make lint in the runner-linters:2.12.2-go1.26.5 image this branch's go:lint job uses and, with the branch merged into main, in the runner-linters:2.14.0-go1.26.10 image main uses since !7681 (merged) (0 issues. in both, test directives included).

Test output on main

With the code from main at 5d62e0cc:

--- FAIL: TestCacheExtractorRemoteServerRejectsPartialContent
    --- FAIL: .../no_archive_on_disk
        Error:      An error is expected but got nil.
        Messages:   a 206 to a GET without Range should fail the download
    --- PASS: .../archive_on_disk_and_up_to_date

The reproduction from #40111 (closed) at the same commit, which runs the download twice:

first run: err=<nil> archive=1 bytes of 159
level=info msg="archive.zip is up to date"
second run: err=<nil> archive=1 bytes, 2 requests in all

With this MR, at eb3d336e:

first run: err=received: 206 Partial Content, but a download without Range needs 200 OK with the whole object; a caching proxy between the runner and the cache server may be replaying a stored partial response archive=0 bytes of 159
second run: err=received: 206 Partial Content, but a download without Range needs 200 OK with the whole object; a caching proxy between the runner and the cache server may be replaying a stored partial response archive=0 bytes, 2 requests in all
ok  gitlab.com/gitlab-org/gitlab-runner/commands/helpers

Environment: Linux amd64, Go 1.26.5 (from go.mod).

What are the relevant issue numbers?

Closes #40111 (closed)