fix(packages): verify the checksum of the newest file with the same name

José M. Requena Plensrequested to merge
gitlab-community/gitlab-org/cli:jmrp-packages-download-newest-checksum into main

Description

glab packages download verifies the file it downloads against the SHA256 stored in the registry, and fetchChecksum took the first file in the package whose name matched (download.go#L308-312). The package files API lists files by ID ascending unless asked otherwise (package_files.rb#L45-63), so that was the oldest upload with that name. The generic download route serves the newest one: it resolves the file with Packages::PackageFileFinder#execute! (generic_packages.rb#L168-169), which returns package_files.last! on a relation with no order, the file with the highest ID (package_file_finder.rb#L16-24).

Generic packages accept a second file with the same name unless duplicates are disabled for the namespace (create_package_file_service.rb#L53-55), and they are allowed by default. So once a file is uploaded again with different contents, every verified download of it fails with a checksum mismatch, and only --no-verify gets past it.

This MR makes fetchChecksum verify against the matching file with the highest ID, which is the file the download route returns, whatever order the list comes in. Several files with one name are a normal state for a generic package rather than an ambiguity, since the download route itself picks the newest, so the command verifies that file instead of failing. A package with one file per name behaves as before.

GitLab 18.6 and later also send the served file's SHA256 in an X-Checksum-SHA256 header on the download route. I kept this change to the lookup, because older instances do not send that header; reading it could be a follow-up.

Related Issues

Closes #8615 (closed)

How has this been tested?

Two cases in Test_PackagesDownload, with a mockChecksumLookupFiles helper that the existing mockChecksumLookup now calls:

  • "Republished file verifies against the most recent upload" lists two app.zip files oldest first, plus a file with another name and a higher ID. Without the change it fails with the old upload's checksum; with it, it passes.
  • "Republished file verifies against the highest ID in any list order" lists the newest app.zip first. It passes with and without the change, and fails if the loop takes the last match in list order instead of the highest ID.

Taking the first match, dropping the file name check, taking the last match in list order, or taking the lowest ID each make one of the two cases fail.

End to end, against GitLab CE 19.4.2 in Docker: two uploads of app.zip with different contents to my-package 1.0.0 in a test project, then glab packages download --name my-package --version 1.0.0 --filename app.zip -R <project> built from main at b3bf6ccb, the commit this branch is based on, and from this branch. The main build fails with the first upload's checksum and saves nothing; this branch saves the second upload, which is what the download route served.

With Go 1.27.2 and golangci-lint 2.14.0: go test -race ./... passes, golangci-lint run reports 0 issues, golangci-lint fmt --diff, go run ./scripts/comment-overlap -threshold 1.0 -max-findings 0 ./cmd/... ./internal/... and make check-embed check-args are clean, make gen-docs and make generate leave the tree unchanged, and the commit message and the title pass commitlint with scripts/commit-lint/.commitlintrc.js.

Test output without and with the change

Without the change to download.go:

--- FAIL: Test_PackagesDownload (0.01s)
    --- FAIL: Test_PackagesDownload/Republished_file_verifies_against_the_most_recent_upload (0.00s)
        download_test.go:233:
            Error:      Received unexpected error:
                        checksum verification failed for app.zip: expected checksum_of_the_old_upload, got 185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969
FAIL	gitlab.com/gitlab-org/cli/internal/commands/packages/download

With it:

    --- PASS: Test_PackagesDownload/Republished_file_verifies_against_the_most_recent_upload (0.00s)
    --- PASS: Test_PackagesDownload/Republished_file_verifies_against_the_highest_ID_in_any_list_order (0.00s)
ok  	gitlab.com/gitlab-org/cli/internal/commands/packages/download

Commands: go test ./internal/commands/packages/download/ -run Test_PackagesDownload -count=1, with -v for the run with the change.

End-to-end run against GitLab CE 19.4.2

The first upload has SHA256 bee1b424..., the second e0c635c7.... After both, the package files API listed these files (id, file_name and file_sha256 of each), and the download route served the second:

GET /api/v4/projects/:id/packages/:package_id/package_files
{"id":1,"file_name":"app.zip","file_sha256":"bee1b424c98e8e095f028278a142ae42ea3c64d5a7559cd9e1a912ff8f81976b"}
{"id":2,"file_name":"app.zip","file_sha256":"e0c635c75a0b831e39b8de86ff153b143aa7b33a60af597c4b6768c821222af5"}

GET /api/v4/projects/:id/packages/generic/my-package/1.0.0/app.zip
SHA256 of body:    e0c635c75a0b831e39b8de86ff153b143aa7b33a60af597c4b6768c821222af5
X-Checksum-Sha256: e0c635c75a0b831e39b8de86ff153b143aa7b33a60af597c4b6768c821222af5

glab packages download --name my-package --version 1.0.0 --filename app.zip -R root/pkg, built from main, exit status 1, no file saved:

ERROR

Checksum verification failed for app.zip: expected bee1b424c98e8e095f028278a142ae42ea3c64d5a7559cd9e1a912ff8f81976b,
got e0c635c75a0b831e39b8de86ff153b143aa7b33a60af597c4b6768c821222af5.

The same command built from this branch, exit status 0:

Downloaded package file to 'app.zip' (Package: my-package, Version: 1.0.0)

sha256sum app.zip on the file it saved:

e0c635c75a0b831e39b8de86ff153b143aa7b33a60af597c4b6768c821222af5  app.zip

Merge request reports