Clean up container virtual registry uploads on digest dedup

What does this MR do and why?

Fixes two problems in container virtual registry cache revalidation. Both are behind the container_virtual_registry_cache_revalidation_fixes feature flag (gitlab_com_derisk, off by default).

  • Quoted ETags never match. The upload callback stores the upstream ETag with its quotes stripped, but revalidation compares it with the raw header. Upstreams that send a quoted ETag, as Docker Hub does, never match, so every stale entry is fetched again. The comparison now strips quotes from the upstream ETag too.
  • Digest dedup leaves the upload behind. When the upload callback finds an existing entry with the same digest, it returns that entry but leaves the new object and its pending direct upload behind, and doesn't mark the entry as checked. It now marks the entry as checked, deletes the new object, and removes its pending direct upload. Digest-addressed content never changes, so the existing entry is as fresh as the upload.
Details
  • The delete reuses ObjectStorage::PendingDirectUpload#delete, the same code the stale upload cleanup runs.
  • It only runs when a pending direct upload exists for the upload, so temp uploads (which Workhorse cleans up itself) are left alone.
  • If the delete fails, the error is tracked and the callback still succeeds. The stale upload cleanup removes what is left.
  • Known edge: the pull can find the entry under one upstream and fetch from another. The dedup lookup is scoped to the upstream that served the upload, so that case is not cleaned up.
  • No conflicts with !259434, which refactors the parent class without changing behavior.

References

  • Related to #631818 (closed) (section "Second problem: revalidation never matches quoted ETags")

How to set up and validate locally

Covered by service and request specs, including a request spec that runs the pull's authorize step and sends its upload ID back on the callback. Not yet validated end to end against a real upstream.

  1. Enable object storage in GDK (gdk config set object_store.enabled true && gdk reconfigure).

  2. In a Rails console, enable the flag for a top-level group with a container virtual registry:

    Feature.enable(:container_virtual_registry_cache_revalidation_fixes, Group.find_by_full_path('<group>'))
  3. Pull an image by digest through the virtual registry, set its cache entry's upstream_checked_at to a year ago, and pull again.

  4. Check that the second pull is served from cache, and that log/application_json.log shows no Pending direct upload prepared event left without a matching completed or deleted.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Tarun Raja

Merge request reports

Loading
Loading