Container virtual registry: refreshed tag serves old manifest with new digest when using object storage
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
When a cached manifest tag in a container virtual registry is refreshed from upstream, and the upstream tag now points to a different digest, the cache entry is updated with the new upstream_etag and digest, but it still points at the old manifest file in object storage. GitLab then returns the new digest in Docker-Content-Digest alongside the old manifest body, and BuildKit fails with:
failed commit on ref "manifest-sha256:<new>": unexpected commit digest sha256:<old>, expected sha256:<new>: failed preconditionThe tag stays broken until its cache entry is deleted. This affects any upstream with mutable tags (reported with Docker Hardened Images), and only when the dependency_proxy object storage is enabled.
Reported in ZD#756856 (Self-Managed Ultimate customer, 19.4.1-ee, Omnibus, AWS S3 with IAM profile, proxy_download = true).
Steps to reproduce
Reproduced on 19.4.1-ee (Omnibus) with consolidated object storage (AWS provider, S3-compatible endpoint, proxy_download = true, bucket configured for dependency_proxy). The upstream must return an Etag header for manifests. Docker Hub and the GitLab container registry do; ttl.sh does not.
- Create a container virtual registry with an upstream whose tag you can update. I used a public project in the same instance's container registry (
https://registry-<ip>.nip.io), withcache_validity_hours: 1. - Push image v1 to
upstream-images/haproxy-like:3.2-dev, using--provenance=false. - Pull through the virtual registry:
FROM <host>/virtual_registries/container/<id>/root/upstream-images/haproxy-like:3.2-dev. This works and the entry is created correctly. - Push image v2 to the same tag upstream.
- Wait for the cache validity to expire. To speed this up:
entry.update_column(:upstream_checked_at, 2.hours.ago). - Pull again. This request is streamed from upstream and succeeds, and the cache entry is updated.
- Pull again with an empty client content store, for example a new
docker buildx create --driver docker-containerbuilder. A builder that already has the v2 manifest locally only sendsHEADand doesn't hit the bug.
What is the current bug behavior?
After step 6, the entry has the v2 digest but the v1 file:
| After first pull | After refresh | |
|---|---|---|
upstream_etag / digest |
sha256:189e1dc7… (v1) |
sha256:b9f645e9… (v2) |
object_storage_key |
…/upstream/2/cache/entry/65/63/6fa21a… |
unchanged |
SHA256 of entry.file.read |
sha256:189e1dc7… |
sha256:189e1dc7… (still v1) |
A new object …/upstream/2/cache/entry/e0/73/e57b33… was written to the bucket at refresh time, and no record references it.
A direct request shows the mismatch:
GET /v2/virtual_registries/container/2/root/upstream-images/haproxy-like/manifests/3.2-dev
HTTP/2 200
docker-content-digest: sha256:b9f645e9… (v2)
body sha256: sha256:189e1dc7… (v1)The by-digest request GET …/manifests/sha256:b9f645e9… is also answered from the same entry, because for_digest matches the updated digest.
Every pull from a clean builder fails from then on. Retrying doesn't help, and neither does waiting out the cache validity period (see the second problem below). Deleting the entry (DELETE /api/v4/virtual_registries/container/cache_entries/:id) fixes it until the next upstream update.
What is the expected correct behavior?
After a refresh, the entry's file, digest and upstream_etag all describe the same manifest, and the replaced object is cleaned up.
Relevant logs and/or screenshots
BuildKit, from a fresh builder after the refresh:
#4 ERROR: failed commit on ref "manifest-sha256:b9f645e9ea6ad9604e1f499fd41f40c92829c618c3594bec6a9c5add28e58d36": unexpected commit digest sha256:189e1dc7509cc94fc2591ff1b2ac87f6de151fdef4fef7b6f7f5decdb2356813, expected sha256:b9f645e9ea6ad9604e1f499fd41f40c92829c618c3594bec6a9c5add28e58d36: failed preconditionCustomer's error, with a DHI multi-arch index:
ERROR: failed to solve: gitlab.example.com/virtual_registries/container/1/haproxy:3.2-dev: failed commit on ref "index-sha256:8aca3b91…": unexpected commit digest sha256:2833e9eb…, expected sha256:8aca3b91…: failed preconditionproduction_json.log during the refresh and the failing pull:
06:06:07 HEAD 200 2/root/upstream-images/haproxy-like/manifests/3.2-dev
06:06:07 POST 200 2/root/upstream-images/haproxy-like/manifests/3.2-dev/upload
06:06:07 GET 200 2/root/upstream-images/haproxy-like/manifests/sha256:b9f645e9…Root cause
- The upload authorization tells Workhorse to upload directly to a new, random final path every time (
container_controller.rb#L212-221, usingUpstream#object_storage_key, which hashesSecureRandom.uuid). BaseCreateOrUpdateCacheEntriesServicecallscreate_or_update_by!, which updates the existing row for the samerelative_pathand passes the newfile(base_create_or_update_cache_entries_service.rb#L25-43).ObjectStorage::Concern#cache_remote_file!writes the new final path intoobject_storage_key(object_storage.rb#L552-556, viaentry_uploader.rb#L39-41), andstore!skips the copy because the file is already at its final path.object_storage_keyisattr_readonly(cache/entry.rb#L43), andraise_on_assign_to_attr_readonly = false(config/application.rb#L43), so the new key is dropped from theUPDATEwithout an error. Everything else in the update (upstream_etag,digest,file_sha1,size,upstream_checked_at) is saved.- On read, the uploader's
store_dircomes from the persisted (old)object_storage_key, so the old object is served, whilesend_cached_filesetsDocker-Content-Digestfrom the newupstream_etag(container_controller.rb#L166).
With local storage there's no direct upload to a final path. CarrierWave stores the file under the existing object_storage_key, replacing the old file, so local storage isn't affected. Other formats that share BaseCreateOrUpdateCacheEntriesService and Cache::Entry (Maven, npm) probably have the same stale file problem on refresh. I haven't checked them.
Second problem: revalidation never matches quoted ETags
upload_params stores the upstream ETag with the quotes removed (sanitize_etag, container_controller.rb#L269-271). cache_response_still_valid? then compares that stored value with the raw upstream header (handle_file_request_service.rb#L45-50):
upstream HEAD etag header = "\"sha256:b9f645e9…\""
stored upstream_etag = "sha256:b9f645e9…"
equal? falseSo for upstreams that quote their ETags (the Docker distribution registry does), a stale entry is never revalidated. It's downloaded again on every expiry. Combined with the first problem, each download leaves another orphaned object in the bucket (7 manifest objects for one tag in this test), and the entry keeps serving the old file.
Output of checks
Reproduced on a test instance, Self-Managed 19.4.1-ee.
Results of GitLab environment info
Expand for output related to GitLab environment info
System: Ubuntu 24.04 Ruby Version: 3.3.12 Redis Version: 7.2.16 Sidekiq Version:7.3.9 GitLab Version: 19.4.1-ee Revision: 26212baacad DB Adapter: PostgreSQL DB Version: 17.11 GitLab Shell: 14.57.3 Gitaly: 19.4.1-g7985a4e Object storage: consolidated form, provider AWS, S3-compatible endpoint (SeaweedFS), path_style, proxy_download = true, dependency_proxy bucket set Customer: 19.4.1-ee, AWS S3, use_iam_profile, proxy_download = true
Possible fixes
These are suggestions and haven't been tested.
- Don't update an existing entry's file in place when the upload went to a new final path. Either create a new
defaultentry and move the old one topending_destruction(that status already exists for cleanup), or allowobject_storage_keyto change on this update path and delete the previous object. - Compare normalized ETags in
cache_response_still_valid?, for example by applying the same quote stripping toresponse.headers['etag']. - Add a spec for updating an existing entry with a direct-uploaded
UploadedFile(remote_idset) that checksentry.reload.file.readreturns the new content. The current "updates it" example increate_or_update_service_spec.rbuses local storage and only checks metadata.
Workaround
Delete the affected tag's cache entry after the upstream tag changes: GET /api/v4/virtual_registries/container/upstreams/:id/cache_entries?search=<image> to find the …/manifests/<tag> entry, then DELETE /api/v4/virtual_registries/container/cache_entries/:id. Clearing the whole upstream cache also works.