Container virtual registry: refreshed tag serves old manifest with new digest when using object storage

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When a cached manifest tag in a container virtual registry is refreshed from upstream, and the upstream tag now points to a different digest, the cache entry is updated with the new upstream_etag and digest, but it still points at the old manifest file in object storage. GitLab then returns the new digest in Docker-Content-Digest alongside the old manifest body, and BuildKit fails with:

failed commit on ref "manifest-sha256:<new>": unexpected commit digest sha256:<old>, expected sha256:<new>: failed precondition

The tag stays broken until its cache entry is deleted. This affects any upstream with mutable tags (reported with Docker Hardened Images), and only when the dependency_proxy object storage is enabled.

Reported in ZD#756856 (Self-Managed Ultimate customer, 19.4.1-ee, Omnibus, AWS S3 with IAM profile, proxy_download = true).

Steps to reproduce

Reproduced on 19.4.1-ee (Omnibus) with consolidated object storage (AWS provider, S3-compatible endpoint, proxy_download = true, bucket configured for dependency_proxy). The upstream must return an Etag header for manifests. Docker Hub and the GitLab container registry do; ttl.sh does not.

  1. Create a container virtual registry with an upstream whose tag you can update. I used a public project in the same instance's container registry (https://registry-<ip>.nip.io), with cache_validity_hours: 1.
  2. Push image v1 to upstream-images/haproxy-like:3.2-dev, using --provenance=false.
  3. Pull through the virtual registry: FROM <host>/virtual_registries/container/<id>/root/upstream-images/haproxy-like:3.2-dev. This works and the entry is created correctly.
  4. Push image v2 to the same tag upstream.
  5. Wait for the cache validity to expire. To speed this up: entry.update_column(:upstream_checked_at, 2.hours.ago).
  6. Pull again. This request is streamed from upstream and succeeds, and the cache entry is updated.
  7. Pull again with an empty client content store, for example a new docker buildx create --driver docker-container builder. A builder that already has the v2 manifest locally only sends HEAD and doesn't hit the bug.

What is the current bug behavior?

After step 6, the entry has the v2 digest but the v1 file:

After first pull After refresh
upstream_etag / digest sha256:189e1dc7… (v1) sha256:b9f645e9… (v2)
object_storage_key …/upstream/2/cache/entry/65/63/6fa21a… unchanged
SHA256 of entry.file.read sha256:189e1dc7… sha256:189e1dc7… (still v1)

A new object …/upstream/2/cache/entry/e0/73/e57b33… was written to the bucket at refresh time, and no record references it.

A direct request shows the mismatch:

GET /v2/virtual_registries/container/2/root/upstream-images/haproxy-like/manifests/3.2-dev
HTTP/2 200
docker-content-digest: sha256:b9f645e9…   (v2)
body sha256:           sha256:189e1dc7…   (v1)

The by-digest request GET …/manifests/sha256:b9f645e9… is also answered from the same entry, because for_digest matches the updated digest.

Every pull from a clean builder fails from then on. Retrying doesn't help, and neither does waiting out the cache validity period (see the second problem below). Deleting the entry (DELETE /api/v4/virtual_registries/container/cache_entries/:id) fixes it until the next upstream update.

What is the expected correct behavior?

After a refresh, the entry's file, digest and upstream_etag all describe the same manifest, and the replaced object is cleaned up.

Relevant logs and/or screenshots

BuildKit, from a fresh builder after the refresh:

#4 ERROR: failed commit on ref "manifest-sha256:b9f645e9ea6ad9604e1f499fd41f40c92829c618c3594bec6a9c5add28e58d36": unexpected commit digest sha256:189e1dc7509cc94fc2591ff1b2ac87f6de151fdef4fef7b6f7f5decdb2356813, expected sha256:b9f645e9ea6ad9604e1f499fd41f40c92829c618c3594bec6a9c5add28e58d36: failed precondition

Customer's error, with a DHI multi-arch index:

ERROR: failed to solve: gitlab.example.com/virtual_registries/container/1/haproxy:3.2-dev: failed commit on ref "index-sha256:8aca3b91…": unexpected commit digest sha256:2833e9eb…, expected sha256:8aca3b91…: failed precondition

production_json.log during the refresh and the failing pull:

06:06:07 HEAD 200 2/root/upstream-images/haproxy-like/manifests/3.2-dev
06:06:07 POST 200 2/root/upstream-images/haproxy-like/manifests/3.2-dev/upload
06:06:07 GET  200 2/root/upstream-images/haproxy-like/manifests/sha256:b9f645e9…

Root cause

  1. The upload authorization tells Workhorse to upload directly to a new, random final path every time (container_controller.rb#L212-221, using Upstream#object_storage_key, which hashes SecureRandom.uuid).
  2. BaseCreateOrUpdateCacheEntriesService calls create_or_update_by!, which updates the existing row for the same relative_path and passes the new file (base_create_or_update_cache_entries_service.rb#L25-43).
  3. ObjectStorage::Concern#cache_remote_file! writes the new final path into object_storage_key (object_storage.rb#L552-556, via entry_uploader.rb#L39-41), and store! skips the copy because the file is already at its final path.
  4. object_storage_key is attr_readonly (cache/entry.rb#L43), and raise_on_assign_to_attr_readonly = false (config/application.rb#L43), so the new key is dropped from the UPDATE without an error. Everything else in the update (upstream_etag, digest, file_sha1, size, upstream_checked_at) is saved.
  5. On read, the uploader's store_dir comes from the persisted (old) object_storage_key, so the old object is served, while send_cached_file sets Docker-Content-Digest from the new upstream_etag (container_controller.rb#L166).

With local storage there's no direct upload to a final path. CarrierWave stores the file under the existing object_storage_key, replacing the old file, so local storage isn't affected. Other formats that share BaseCreateOrUpdateCacheEntriesService and Cache::Entry (Maven, npm) probably have the same stale file problem on refresh. I haven't checked them.

Second problem: revalidation never matches quoted ETags

upload_params stores the upstream ETag with the quotes removed (sanitize_etag, container_controller.rb#L269-271). cache_response_still_valid? then compares that stored value with the raw upstream header (handle_file_request_service.rb#L45-50):

upstream HEAD etag header = "\"sha256:b9f645e9…\""
stored upstream_etag      = "sha256:b9f645e9…"
equal? false

So for upstreams that quote their ETags (the Docker distribution registry does), a stale entry is never revalidated. It's downloaded again on every expiry. Combined with the first problem, each download leaves another orphaned object in the bucket (7 manifest objects for one tag in this test), and the entry keeps serving the old file.

Output of checks

Reproduced on a test instance, Self-Managed 19.4.1-ee.

Results of GitLab environment info

Expand for output related to GitLab environment info
System:         Ubuntu 24.04
Ruby Version:   3.3.12
Redis Version:  7.2.16
Sidekiq Version:7.3.9
GitLab Version: 19.4.1-ee
Revision:       26212baacad
DB Adapter:     PostgreSQL
DB Version:     17.11
GitLab Shell:   14.57.3
Gitaly:         19.4.1-g7985a4e

Object storage: consolidated form, provider AWS, S3-compatible endpoint (SeaweedFS), path_style, proxy_download = true, dependency_proxy bucket set
Customer: 19.4.1-ee, AWS S3, use_iam_profile, proxy_download = true

Possible fixes

These are suggestions and haven't been tested.

  • Don't update an existing entry's file in place when the upload went to a new final path. Either create a new default entry and move the old one to pending_destruction (that status already exists for cleanup), or allow object_storage_key to change on this update path and delete the previous object.
  • Compare normalized ETags in cache_response_still_valid?, for example by applying the same quote stripping to response.headers['etag'].
  • Add a spec for updating an existing entry with a direct-uploaded UploadedFile (remote_id set) that checks entry.reload.file.read returns the new content. The current "updates it" example in create_or_update_service_spec.rb uses local storage and only checks metadata.

Workaround

Delete the affected tag's cache entry after the upstream tag changes: GET /api/v4/virtual_registries/container/upstreams/:id/cache_entries?search=<image> to find the …/manifests/<tag> entry, then DELETE /api/v4/virtual_registries/container/cache_entries/:id. Clearing the whole upstream cache also works.

Edited by 🤖 GitLab Bot 🤖