Fix stale container virtual registry cache on refetch
What does this MR do and why?
Fixes the container virtual registry serving stale cached files after a refetch. This is the container version of the Maven fix in !259434.
The bug: a cache entry's object_storage_key is read-only. When a stale entry was refetched from the upstream, Workhorse uploaded to that same final path, so the new bytes never replaced the cached file. Two Ultimate customers hit this.
The fix:
- Refetch detection:
HandleFileRequestService#overwrite?setsoverwrite: truewhen the picked upstream already has an entry for the path. It's set server-side and travels in the Workhorse-signed payload. - Temp upload: with
overwrite, the controller authorizes a temp upload instead of the final store path. - Copy under a lease: the cache entry service copies the temp upload onto the existing entry's key under an exclusive lease. A concurrent refetch of the same file skips. If the entry was removed mid-upload, it returns
:cache_entry_removed. - Feature flag: the overwrite path is behind
container_virtual_registry_refetch_overwrite(gitlab_com_derisk, off by default). With it off, both the authorize step and the cache entry service follow the master path. Rollout: #639178 - Shared helper:
BaseCreateOrUpdateCacheEntriesService#entry_updatesis the same extraction as in !259434. Either MR can merge first.git merge-treewith the Maven branch is clean. - Digest dedup for blobs runs first, as before.
Closes #632449 (closed).
How to validate
The specs cover copy-then-update, lease taken, same-bytes skip, entry removed, permission denied, digest dedup, a first-fetch round trip, and a file that moved to a lower-priority upstream. Six of the new service specs fail on master's code.
- 389 examples across the container controller, the container services, the Maven and npm create/update services, and the container cache entries API pass with dependency proxy object storage off, as in CI.
- Rubocop is clean.
Follow-ups, not in this MR
- About 70 lines of refetch logic now exist in both the Maven and container services. Pull them into a shared module after both MRs merge.
- Objects already orphaned in production get cleaned up with the Maven follow-up, as planned on the issue.
Database
No schema changes. The refetch path runs the same query shapes that create_or_update_by! already runs on master: a lookup by relative path, then update! on the loaded row. The table (virtual_registries_container_cache_remote_entries) is hash-partitioned on group_id into 16 partitions.
How I got the plans: SQL captured from the code path on a local GDK. Production holds a few thousand rows, so I seeded 50,000 entries across five upstreams in group 9970 on the gitlab-production-main Database Lab clone, ran ANALYZE, then explained. The clone was reset afterwards. Timings are warm, since the rows were just seeded.
1. Find the existing entry (new call sites)
Runs in HandleFileRequestService#overwrite? and again under the lease in CreateOrUpdateService#existing_entry_response.
SELECT "virtual_registries_container_cache_remote_entries".*
FROM "virtual_registries_container_cache_remote_entries"
WHERE "virtual_registries_container_cache_remote_entries"."upstream_id" = <upstream_id>
AND "virtual_registries_container_cache_remote_entries"."group_id" = <group_id>
AND "virtual_registries_container_cache_remote_entries"."status" = 0
AND "virtual_registries_container_cache_remote_entries"."relative_path" = '<relative_path>'
LIMIT 1Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187713
(Index Scan on idx_uniq_vreg_cont_cache_remote_upstream_rel_path, one partition, 1 row, 0.15 ms, buffers: 7 hit / 0 read)
2. Update the refetched entry (Danger thread)
existing_entry.update! in CreateOrUpdateService#refetch. Single row by primary key. file_store is only in the SET list when the entry moves from local to object storage.
UPDATE "virtual_registries_container_cache_remote_entries"
SET "updated_at" = '<now>', "upstream_checked_at" = '<now>', "file_sha1" = '<sha1>',
"upstream_etag" = '<etag>', "content_type" = '<content_type>', "digest" = '<digest>',
"size" = <size>, "file_store" = 2
WHERE "virtual_registries_container_cache_remote_entries"."group_id" = <group_id>
AND "virtual_registries_container_cache_remote_entries"."iid" = <iid>Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187716 (Index Scan on the partition primary key, 1 row, 1.1 ms, buffers: 56 hit / 0 read)
3. Uniqueness validations run by update!
These come from the existing relative_path uniqueness validators on the base and container models. They are not new, but they run as part of the update! above.
Base model validator (scope upstream_id, status, no group_id):
SELECT 1 AS one FROM "virtual_registries_container_cache_remote_entries"
WHERE "virtual_registries_container_cache_remote_entries"."relative_path" = '<relative_path>'
AND NOT ("virtual_registries_container_cache_remote_entries"."group_id" = <group_id>
AND "virtual_registries_container_cache_remote_entries"."iid" = <iid>)
AND "virtual_registries_container_cache_remote_entries"."upstream_id" = <upstream_id>
AND "virtual_registries_container_cache_remote_entries"."status" = 0
LIMIT 1Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187714
(Index Scans on the upstream_id indexes in all 16 partitions, since there is no group_id to prune on; 0 rows, 10.6 ms, buffers: 120 hit / 11 read)
Container model validator (adds group_id):
SELECT 1 AS one FROM "virtual_registries_container_cache_remote_entries"
WHERE "virtual_registries_container_cache_remote_entries"."relative_path" = '<relative_path>'
AND NOT ("virtual_registries_container_cache_remote_entries"."group_id" = <group_id>
AND "virtual_registries_container_cache_remote_entries"."iid" = <iid>)
AND "virtual_registries_container_cache_remote_entries"."upstream_id" = <upstream_id>
AND "virtual_registries_container_cache_remote_entries"."status" = 0
AND "virtual_registries_container_cache_remote_entries"."group_id" = <group_id>
LIMIT 1Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/79727/commands/187715
(Index Scan on idx_uniq_vreg_cont_cache_remote_upstream_rel_path, one partition, 0 rows, 0.1 ms, buffers: 7 hit / 0 read)
bump_downloads_count sends no SQL here.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.