Discard rejected package uploads to a final store path

What does this MR do and why?

Maven, Generic and Terraform module uploads are authorized to a final store path. The authorize request adds a pending direct upload entry, and Workhorse uploads the file directly to that path. When the file is stored, the entry is removed.

If the upload request rejects the file instead, nothing stores it. Rejections include a duplicate package, a protected package, or a file that is too large. The entry and the uploaded object then stay until the stale upload cleanup reaches them. On GitLab.com, duplicate uploads are now the largest source of these entries (numbers below).

This MR deletes the entry and the object when an upload request ends with its entry still pending:

  • ObjectStorage::PendingDirectUpload.discard deletes the object and the entry, but only if the entry still exists.
  • discarding_unstored_upload in API::Helpers::PackagesHelpers wraps the upload endpoint bodies and calls it in an ensure block, so it runs for every rejection inside the endpoint body after authorization.

Rejections that happen before the endpoint body, in authorize_upload!, authorize_job_token_policies! (Generic) or the before filter, still leave the entry and the object until the stale upload cleanup. Follow-up: https://gitlab.com/gitlab-org/gitlab/-/work_items/639241

Successful uploads are unaffected: storing the file already removed the entry, so discard does nothing. Errors raised while discarding are tracked and do not change the response.

The change is behind the packages_discard_unstored_direct_uploads feature flag (gitlab_com_derisk, disabled by default).

Why deleting at the end of the request is safe

Each authorize call generates a new final store path. Workhorse returns it as remote_id in signed upload params that are valid only for that request. If the entry still exists when the request ends, no other request can store that file.

Production numbers (Kibana, Rails logs)

Before !265501 (merged) (24 hours to 2026-10-27): Maven left 454,420 entries never completed, against 453,860 .sha1 and .md5 uploads. Generic left 12,617.

After !265501 (merged) (24 hours to 2026-11-04, about 20:50 UTC):

Maven Count
Pending direct upload prepared 506,799
Pending direct upload completed 404,922
Never completed about 101,900
Upload PUT returning 400 Duplicate package is not allowed 102,293

Generic, 7 days to 2026-11-05: one CI client sent 121,585 upload requests, and 86.6% returned 400 Duplicate package is not allowed.

Rollout baseline: with the flag on, Maven prepared should match completed plus deleted (logged by the upload route, not the stale upload cleanup worker).

References

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

  1. Enable object storage for packages (for example, the GDK MinIO setup). Direct upload is always on for packages (config/object_store_settings.rb).

  2. In the Rails console, enable the feature flag:

    Feature.enable(:packages_discard_unstored_direct_uploads)
  3. In the project's top-level group, go to Settings > Packages and registries and turn off duplicate Generic packages.

  4. Upload a Generic package file:

    curl --header "PRIVATE-TOKEN: <token>" --upload-file ./file.txt \
      "http://gdk.test:3000/api/v4/projects/<project_id>/packages/generic/my_package/1.0.0/file.txt"
  5. Note the count in the Rails console:

    ObjectStorage::PendingDirectUpload.count
  6. Run the same curl command again. It returns 400 Duplicate package is not allowed.

  7. Check the count again. It is the same as before the second upload. With the flag disabled, it goes up by one, and the object stays under @final/ in the packages bucket.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Tarun Raja

Merge request reports

Loading
Loading