Fix stale Maven virtual registry cache on refetch
What does this MR do and why?
Refetching a cached remote Maven file now replaces the bytes the entry actually serves.
Bug: a customer using Maven Central as an upstream got a 5-month-old maven-metadata.xml. Version ranges, LATEST, and RELEASE resolved against an old version list with no error. Their data showed upstream_checked_at from 2026-06-30 and content from 2026-01-26.
Root cause (found by @10io in a functional test): on a refetch, Workhorse uploads the new bytes to a new random object key. The entry's object_storage_key is attr_readonly, so the row keeps pointing at the old object. The row looks fresh. The bytes are old. The new object is orphaned. This affects any refetched remote Maven file, but metadata is the file that changes in practice.
Fix (plan by @10io, with changes from @mkhalifa3's review):
- Authorize: a refetch of the picked upstream's existing remote entry authorizes a temp upload. New entries keep the final-path upload, so large first fetches (up to 5 GB) pay no copy cost.
- Lookup:
HandleFileRequestServiceloads the cache entries once, keyed by upstream, and picks the first one in upstream position order, from the upstream list it already loaded.overwrite?checks the picked upstream in memory. - Copy:
Maven::Cache::Entries::CreateOrUpdateServicedetects the temp upload from the upload itself. Under an exclusive lease on the entry, it looks the entry up once, copies the temp object onto the entry's existing key, then updates the row's metadata. An entry still in local storage moves to object storage, where the copy put the new bytes. The copy runs outside any transaction, and the duration and size are logged. - Key: the key never changes. No
attr_readonlychange, no delete code, no orphans.
Not in this MR:
- npm: the npm virtual registry shares the endpoint but does not pass the flag, so it is unchanged.
- Container: the container virtual registry has the same bug. Tracked in #632449 (closed).
- Digests: !259437 (closed) (revalidate digests of stale metadata) treats a symptom of the same bug. We will close it once this lands.
An earlier version of this MR forced a refetch of stale maven-metadata.xml. That is dropped. ETag revalidation is unchanged.
Timing: an Artifact Central design partner plans a production rollout in 2-3 months.
Why this is safe
- Early failures: failures before the copy (upstream error, size limit) only lose the temp object. The live object is untouched.
- Entry removed mid-refetch: if a purge or cleanup removes the entry while Workhorse streams the upload, the service returns an error. It doesn't create a row that points at a missing object.
- Stale-upload cleanup: no pending direct upload is registered for the live key, so the 3-hour cleanup never targets it.
- No transaction during the copy: the row is updated only after the copy, so no row lock or connection is held while bytes move.
- Copy then update: if the copy succeeds and the row update then fails, the object has new bytes while the row keeps the old digest, until the next refetch. The update is short, so this window is small. We accept it rather than hold a transaction open across the copy.
- Concurrent refetches: an exclusive lease on the entry runs them one at a time. A refetch that can't take the lease skips, because the holder writes the same bytes.
- Statistics: the row is saved with
update!, so the dependency proxy size refresh still runs. - Cleanup: Workhorse deletes the temp object at the end of the request.
Why not overwrite in place or swap the key
- In-place overwrite of the live key: authorize registers a pending direct upload for that key. The 3-hour cleanup deletes it without checking references. Workhorse also deletes a final-path object right away if the upload fails midway. Either could delete the live cached file.
- Key swap: CarrierWave's remove-previous would not delete the old object, because the filename identifier does not change. It needs custom delete code and removing
attr_readonly.
Why no feature flag
This is a bug fix. It only changes how a refetch is stored. There are no schema, UI, or API changes.
Database
The MR adds one scope (for_relative_path) and changes three queries. No schema changes. The SQL below comes from the spec runs.
Plans are from Database Lab (gitlab-production-main, Joe CLI), using the busiest real group, registry, and upstream. Literals are replaced with placeholders. The CLI doesn't give console links, so the Joe command ID is listed with each plan.
1. Remote entry lookup (HandleFileRequestService#remote_cache_entries, new scope for_relative_path). This replaces find_by_relative_path, so it drops LIMIT 1. It returns at most one row per upstream, from the unique index idx_uniq_vreg_mvn_cache_remote_entries_on_upstream_id_rel_path on (upstream_id, group_id, relative_path) WHERE status = 0. A registry has at most 20 upstreams.
SELECT "virtual_registries_packages_maven_cache_remote_entries".*
FROM "virtual_registries_packages_maven_cache_remote_entries"
WHERE "virtual_registries_packages_maven_cache_remote_entries"."status" = 0
AND "virtual_registries_packages_maven_cache_remote_entries"."group_id" = :group_id
AND "virtual_registries_packages_maven_cache_remote_entries"."upstream_id" IN (
SELECT "virtual_registries_packages_maven_upstreams"."id"
FROM "virtual_registries_packages_maven_upstreams"
INNER JOIN "virtual_registries_packages_maven_registry_upstreams"
ON "virtual_registries_packages_maven_upstreams"."id" = "virtual_registries_packages_maven_registry_upstreams"."upstream_id"
WHERE "virtual_registries_packages_maven_registry_upstreams"."registry_id" = :registry_id
ORDER BY "virtual_registries_packages_maven_registry_upstreams"."position" ASC
)
AND "virtual_registries_packages_maven_cache_remote_entries"."relative_path" = :relative_pathPlan (Joe command 532452): Index scan on the remote entries partition (relative_path, group_id), plus index scans on registry upstreams. 1 row, 11.0 ms, 16 buffers. No seq scan. The planner picked the partition index virtual_registries_packages_m_relative_path_object_storage_idx5 on (relative_path, group_id), not idx_uniq_vreg_mvn_cache_remote_entries_on_upstream_id_rel_path. The unique index still keeps it to one row per upstream.
Plan
Nested Loop Semi Join (cost=1.10..13.27 rows=1 width=478) (actual time=7.027..7.030 rows=1 loops=1)
Join Filter: (virtual_registries_packages_maven_upstreams.id = virtual_registries_packages_maven_cache_remote_entries.upstream_id)
Rows Removed by Join Filter: 1
Buffers: shared hit=11 read=5
I/O Timings: read=6.857 write=0.000
-> Index Scan using virtual_registries_packages_m_relative_path_object_storage_idx5 on gitlab_partitions_static.virtual_registries_packages_maven_cache_remote_entries_05 virtual_registries_packages_maven_cache_remote_entries (cost=0.55..3.57 rows=1 width=478) (actual time=6.015..6.016 rows=1 loops=1)
Index Cond: ((virtual_registries_packages_maven_cache_remote_entries.relative_path = :relative_path) AND (virtual_registries_packages_maven_cache_remote_entries.group_id = :group_id))
Filter: (virtual_registries_packages_maven_cache_remote_entries.status = 0)
Buffers: shared hit=1 read=4
I/O Timings: read=5.933 write=0.000
-> Nested Loop (cost=0.55..9.66 rows=2 width=10) (actual time=1.005..1.008 rows=2 loops=1)
Buffers: shared hit=10 read=1
I/O Timings: read=0.924 write=0.000
-> Index Scan using constraint_vreg_pkgs_mvn_reg_upst_on_unique_regid_pos on public.virtual_registries_packages_maven_registry_upstreams (cost=0.28..4.57 rows=2 width=10) (actual time=0.990..0.991 rows=2 loops=1)
Index Cond: (virtual_registries_packages_maven_registry_upstreams.registry_id = :registry_id)
Buffers: shared hit=5 read=1
I/O Timings: read=0.924 write=0.000
-> Index Only Scan using virtual_registries_packages_maven_upstreams_pkey on public.virtual_registries_packages_maven_upstreams (cost=0.28..2.54 rows=1 width=8) (actual time=0.006..0.006 rows=1 loops=2)
Index Cond: (virtual_registries_packages_maven_upstreams.id = virtual_registries_packages_maven_registry_upstreams.upstream_id)
Heap Fetches: 0
Buffers: shared hit=5
I/O Timings: read=0.000 write=0.000
Settings: effective_cache_size = '472585MB', work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4'
Query ID: 82791385921681723992. Local entry lookup (HandleFileRequestService#local_cache_entries, new scope for_relative_path). Same change as query 1. The unique index idx_uniq_vreg_mvn_cache_local_entries_on_upstream_id_rel_path on (upstream_id, group_id, relative_path) keeps it to one row per upstream. The existing preload(:package_file) query is unchanged.
SELECT "virtual_registries_packages_maven_cache_local_entries".*
FROM "virtual_registries_packages_maven_cache_local_entries"
WHERE "virtual_registries_packages_maven_cache_local_entries"."group_id" = :group_id
AND "virtual_registries_packages_maven_cache_local_entries"."upstream_id" IN (
SELECT "virtual_registries_packages_maven_local_upstreams"."id"
FROM "virtual_registries_packages_maven_local_upstreams"
INNER JOIN "virtual_registries_packages_maven_registry_upstreams"
ON "virtual_registries_packages_maven_local_upstreams"."id" = "virtual_registries_packages_maven_registry_upstreams"."local_upstream_id"
WHERE "virtual_registries_packages_maven_registry_upstreams"."registry_id" = :registry_id
ORDER BY "virtual_registries_packages_maven_registry_upstreams"."position" ASC
)
AND "virtual_registries_packages_maven_cache_local_entries"."relative_path" = :relative_pathPlan (Joe command 532454): Index scan on idx_uniq_vreg_mvn_cache_local_entries_on_upstream_id_rel_path (partition), plus index scans on registry upstreams. 1 row, 6.4 ms, 14 buffers. No seq scan.
Plan
Nested Loop (cost=5.19..46.21 rows=1 width=136) (actual time=4.138..4.141 rows=1 loops=1)
Buffers: shared hit=12 read=2
I/O Timings: read=3.939 write=0.000
-> HashAggregate (cost=4.78..4.90 rows=12 width=8) (actual time=0.866..0.868 rows=1 loops=1)
Group Key: virtual_registries_packages_maven_local_upstreams.id
Batches: 1 Memory Usage: 24kB
Buffers: shared hit=6 read=1
I/O Timings: read=0.761 write=0.000
-> Sort (cost=4.60..4.63 rows=12 width=10) (actual time=0.859..0.860 rows=1 loops=1)
Sort Key: virtual_registries_packages_maven_registry_upstreams."position"
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=6 read=1
I/O Timings: read=0.761 write=0.000
-> Merge Join (cost=0.28..4.38 rows=12 width=10) (actual time=0.815..0.819 rows=1 loops=1)
Merge Cond: (virtual_registries_packages_maven_local_upstreams.id = virtual_registries_packages_maven_registry_upstreams.local_upstream_id)
Buffers: shared hit=3 read=1
I/O Timings: read=0.761 write=0.000
-> Index Only Scan using virtual_registries_packages_maven_local_upstreams_pkey on public.virtual_registries_packages_maven_local_upstreams (cost=0.14..3.74 rows=40 width=8) (actual time=0.005..0.009 rows=8 loops=1)
Heap Fetches: 0
Buffers: shared hit=2
I/O Timings: read=0.000 write=0.000
-> Index Scan using idx_mvn_reg_upstreams_on_local_upstream_and_registry_ids on public.virtual_registries_packages_maven_registry_upstreams (cost=0.14..4.86 rows=12 width=10) (actual time=0.800..0.802 rows=1 loops=1)
Index Cond: (virtual_registries_packages_maven_registry_upstreams.registry_id = :registry_id)
Buffers: shared hit=1 read=1
I/O Timings: read=0.761 write=0.000
-> Index Scan using virtual_registries_packages__upstream_id_group_id_relativ_idx48 on gitlab_partitions_static.virtual_registries_packages_maven_cache_local_entries_10 virtual_registries_packages_maven_cache_local_entries (cost=0.41..3.43 rows=1 width=136) (actual time=3.267..3.268 rows=1 loops=1)
Index Cond: ((virtual_registries_packages_maven_cache_local_entries.upstream_id = virtual_registries_packages_maven_local_upstreams.id) AND (virtual_registries_packages_maven_cache_local_entries.group_id = :group_id) AND (virtual_registries_packages_maven_cache_local_entries.relative_path = :relative_path))
Buffers: shared hit=6 read=1
I/O Timings: read=3.178 write=0.000
Settings: effective_cache_size = '472585MB', work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4'
Query ID: 81582411190793194343. Refetch lookup (Maven::Cache::Entries::CreateOrUpdateService#existing_entry_response). Runs only on a refetch, under the exclusive lease. Same unique index as query 1.
SELECT "virtual_registries_packages_maven_cache_remote_entries".*
FROM "virtual_registries_packages_maven_cache_remote_entries"
WHERE "virtual_registries_packages_maven_cache_remote_entries"."upstream_id" = :upstream_id
AND "virtual_registries_packages_maven_cache_remote_entries"."group_id" = :group_id
AND "virtual_registries_packages_maven_cache_remote_entries"."status" = 0
AND "virtual_registries_packages_maven_cache_remote_entries"."relative_path" = :relative_path
LIMIT 1Plan (Joe command 532456): Index scan on idx_uniq_vreg_mvn_cache_remote_entries_on_upstream_id_rel_path (partition). 1 row, 3.8 ms, 7 buffers.
Plan
Limit (cost=0.42..3.44 rows=1 width=478) (actual time=1.014..1.015 rows=1 loops=1)
Buffers: shared hit=6 read=1
I/O Timings: read=0.929 write=0.000
-> Index Scan using virtual_registries_packages_m_upstream_id_group_id_relativ_idx5 on gitlab_partitions_static.virtual_registries_packages_maven_cache_remote_entries_05 virtual_registries_packages_maven_cache_remote_entries (cost=0.42..3.44 rows=1 width=478) (actual time=1.013..1.014 rows=1 loops=1)
Index Cond: ((virtual_registries_packages_maven_cache_remote_entries.upstream_id = :upstream_id) AND (virtual_registries_packages_maven_cache_remote_entries.group_id = :group_id) AND (virtual_registries_packages_maven_cache_remote_entries.relative_path = :relative_path))
Buffers: shared hit=6 read=1
I/O Timings: read=0.929 write=0.000
Settings: jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB', work_mem = '230MB'
Query ID: 78153370544645489774. Refetch update (Maven::Cache::Entries::CreateOrUpdateService#refetch). Updates one row by primary key (group_id, iid). It runs after the object copy, so no transaction is open while bytes move. file_store is set only when the entry moves from local storage. The existing uniqueness validation query on update! is unchanged.
UPDATE "virtual_registries_packages_maven_cache_remote_entries"
SET "updated_at" = :now, "upstream_checked_at" = :now, "size" = :size,
"file_sha1" = :sha1, "file_md5" = :md5, "upstream_etag" = :etag,
"content_type" = :content_type, "file_store" = 2
WHERE "virtual_registries_packages_maven_cache_remote_entries"."group_id" = :group_id
AND "virtual_registries_packages_maven_cache_remote_entries"."iid" = :iidPlan (Joe command 532458): Index scan on the partition primary key (group_id, iid). 1 row updated, 14.5 ms, 71 buffers (9 dirtied).
Plan
Update on public.virtual_registries_packages_maven_cache_remote_entries (cost=0.42..3.44 rows=0 width=0) (actual time=12.814..12.814 rows=0 loops=1)
Buffers: shared hit=60 read=11 dirtied=9
WAL: records=8 fpi=7 bytes=43478
I/O Timings: read=11.647 write=0.000
-> Index Scan using virtual_registries_packages_maven_cache_remote_entries_05_pkey on gitlab_partitions_static.virtual_registries_packages_maven_cache_remote_entries_05 virtual_registries_packages_maven_cache_remote_entries_1 (cost=0.42..3.44 rows=1 width=160) (actual time=1.017..1.019 rows=1 loops=1)
Index Cond: ((virtual_registries_packages_maven_cache_remote_entries_1.group_id = :group_id) AND (virtual_registries_packages_maven_cache_remote_entries_1.iid = :iid))
Buffers: shared hit=6 read=1
I/O Timings: read=0.950 write=0.000
Settings: effective_cache_size = '472585MB', work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4'
Query ID: -8972895151046791283References
- Issue: #604701
- Container follow-up: #632449 (closed)
- Artifact Registry parity: gitlab-org/ops/artifact-registry#1519
Screenshots or screen recordings
Not applicable. No UI changes.
How to set up and validate locally
Tests:
Maven::Cache::Entries::CreateOrUpdateServicespec: uses fog-mocked object storage. A temp upload on an existing entry ends atobject_storage_key, the entry serves the new bytes, and the statistics refresh runs. It also covers a failed copy, a failed update, a held lease, a removed entry, a final-path upload, a local-storage entry moving to object storage, and no permission.HandleFileRequestServicespec:overwriteis true for a refetch from the entry's upstream (changed ETag, blank ETag). It is false when a different upstream is picked or the entry is new. The lookup picks the highest-priority upstream's entry.- Request spec: the authorize call is a temp upload when
overwriteis true.
Checked on GDK with MinIO (Rails runner): stored an entry with "old bytes", put a temp object, and saved the refetch. The key did not change. The served bytes were "new bytes" and matched file_sha1. One object sat under the key.
Reproduce locally:
- Enable object storage for the dependency proxy in GDK (MinIO).
- Configure a Maven virtual registry with an upstream.
- Pull a file once.
- In the Rails console, set the entry's
upstream_checked_atto a year ago andupstream_etagto a different value. - Pull it again.
- Check that
entry.reload.file.readmatches the upstream file, that its SHA1 matchesfile_sha1, and that the bucket has one object underentry.object_storage_key.
On master, the object under the key keeps the old bytes and a second object appears.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.