Use the new Cache::Remote::Entry model for container virtual registry (!220483) · Merge requests · GitLab.org / GitLab

What does this MR do and why?

This merge request refactors the virtual registry container cache system by migrating from an old cache entry table to a new "remote entries" table. The changes include:

Main Updates:

Deprecates the old virtual_registries_container_cache_entries table in favor of a new virtual_registries_container_cache_remote_entries table
Updates all code references to use the new remote entry model instead of the old cache entry model
Improves the remote entry model with better validation rules, file storage capabilities, and search functionality

Technical Improvements:

Adds proper authorization checks to GraphQL types
Implements file upload handling and storage management for cached files
Creates new API endpoints that use the updated remote entry structure
Updates cleanup and audit services to work with the new table structure
Uses database constraints and indexing for better data integrity

User Impact: The changes maintain the same functionality for users while providing a more robust backend system for caching container registry data. The migration ensures better performance, data validation, and file management for virtual registry operations.

References

Container: redo partitioning of virtual_registr... (#583726 - closed)

Database Query Plans

Fetch cache entries: !220483 (comment 3044792048)
Fetch a cache entry: !220483 (comment 3044847012)
!220483 (comment 3038631562)
!220483 (comment 3038717823)
!220483 (comment 3038754480)
!220483 (comment 3038822784)
!220483 (comment 3038902132)
!220483 (comment 3038942324)
!220483 (comment 3039071943)

Migration result:

Migration 1

main: == [advisory_lock_connection] object_id: 133460, pg_backend_pid: 88075
main: == 20260130154427 AddVirtualRegistriesContainerCacheRemoteIndex: migrating ====
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_name_exists?(:virtual_registries_container_cache_remote_entries, :idx_vreg_cont_cache_remote_lookup)
main:    -> 0.0009s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_00", [:upstream_id, :group_id, :created_at], {:name=>"index_ed5ccdc4cd", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0015s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_00", [:upstream_id, :group_id, :created_at], {:name=>"index_ed5ccdc4cd", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0039s
main: -- execute("RESET statement_timeout")
main:    -> 0.0005s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_01", [:upstream_id, :group_id, :created_at], {:name=>"index_8d1d9cd2cb", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0012s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_01", [:upstream_id, :group_id, :created_at], {:name=>"index_8d1d9cd2cb", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0027s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_02", [:upstream_id, :group_id, :created_at], {:name=>"index_7e851ceff6", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0011s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_02", [:upstream_id, :group_id, :created_at], {:name=>"index_7e851ceff6", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0022s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_03", [:upstream_id, :group_id, :created_at], {:name=>"index_0586981065", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0011s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_03", [:upstream_id, :group_id, :created_at], {:name=>"index_0586981065", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0026s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_04", [:upstream_id, :group_id, :created_at], {:name=>"index_9eb022c7ea", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0014s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_04", [:upstream_id, :group_id, :created_at], {:name=>"index_9eb022c7ea", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0027s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_05", [:upstream_id, :group_id, :created_at], {:name=>"index_d40c2d7e7d", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0011s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_05", [:upstream_id, :group_id, :created_at], {:name=>"index_d40c2d7e7d", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0023s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_06", [:upstream_id, :group_id, :created_at], {:name=>"index_97ea88d542", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0014s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_06", [:upstream_id, :group_id, :created_at], {:name=>"index_97ea88d542", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0021s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_07", [:upstream_id, :group_id, :created_at], {:name=>"index_6ee4c2760a", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0011s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_07", [:upstream_id, :group_id, :created_at], {:name=>"index_6ee4c2760a", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0023s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_08", [:upstream_id, :group_id, :created_at], {:name=>"index_1b7c5225c3", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0011s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_08", [:upstream_id, :group_id, :created_at], {:name=>"index_1b7c5225c3", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0022s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_09", [:upstream_id, :group_id, :created_at], {:name=>"index_d7a5d3faf2", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0013s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_09", [:upstream_id, :group_id, :created_at], {:name=>"index_d7a5d3faf2", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0023s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_10", [:upstream_id, :group_id, :created_at], {:name=>"index_70a74a41e7", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0011s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_10", [:upstream_id, :group_id, :created_at], {:name=>"index_70a74a41e7", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0027s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_11", [:upstream_id, :group_id, :created_at], {:name=>"index_c03c3c5941", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0012s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_11", [:upstream_id, :group_id, :created_at], {:name=>"index_c03c3c5941", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0025s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_12", [:upstream_id, :group_id, :created_at], {:name=>"index_788d2c6465", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0011s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_12", [:upstream_id, :group_id, :created_at], {:name=>"index_788d2c6465", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0028s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_13", [:upstream_id, :group_id, :created_at], {:name=>"index_53184a1801", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0011s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_13", [:upstream_id, :group_id, :created_at], {:name=>"index_53184a1801", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0022s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_14", [:upstream_id, :group_id, :created_at], {:name=>"index_3db5021697", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0013s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_14", [:upstream_id, :group_id, :created_at], {:name=>"index_3db5021697", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0023s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_15", [:upstream_id, :group_id, :created_at], {:name=>"index_bb408e6818", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0011s
main: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_15", [:upstream_id, :group_id, :created_at], {:name=>"index_bb408e6818", :order=>{:created_at=>:desc}, :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0022s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- add_index(:virtual_registries_container_cache_remote_entries, [:upstream_id, :group_id, :created_at], {:name=>:idx_vreg_cont_cache_remote_lookup, :order=>{:created_at=>:desc}, :where=>"status = 0"})
main:    -> 0.0031s
main: == 20260130154427 AddVirtualRegistriesContainerCacheRemoteIndex: migrated (0.1951s)

main: == [advisory_lock_connection] object_id: 133460, pg_backend_pid: 88075

Migration 2

ci: == [advisory_lock_connection] object_id: 134060, pg_backend_pid: 90677
ci: == 20260130154428 AddRequiringCleanupIdxToVirtualRegistriesContainerCacheRemEntries: migrating
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_name_exists?(:virtual_registries_container_cache_remote_entries, :index_vr_cont_cache_rem_entries_on_requiring_cleanup)
ci:    -> 0.0014s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_00", [:group_id, :iid, :downloaded_at], {:name=>"index_f25c287606", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0022s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0002s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_00", [:group_id, :iid, :downloaded_at], {:name=>"index_f25c287606", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0052s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0002s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_01", [:group_id, :iid, :downloaded_at], {:name=>"index_9f804f9636", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0013s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_01", [:group_id, :iid, :downloaded_at], {:name=>"index_9f804f9636", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0022s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_02", [:group_id, :iid, :downloaded_at], {:name=>"index_7cfe703df0", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0016s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_02", [:group_id, :iid, :downloaded_at], {:name=>"index_7cfe703df0", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0029s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_03", [:group_id, :iid, :downloaded_at], {:name=>"index_3d578831b2", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0016s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_03", [:group_id, :iid, :downloaded_at], {:name=>"index_3d578831b2", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0023s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_04", [:group_id, :iid, :downloaded_at], {:name=>"index_db4f6a9088", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0014s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_04", [:group_id, :iid, :downloaded_at], {:name=>"index_db4f6a9088", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0026s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_05", [:group_id, :iid, :downloaded_at], {:name=>"index_230e63a655", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0015s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_05", [:group_id, :iid, :downloaded_at], {:name=>"index_230e63a655", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0021s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_06", [:group_id, :iid, :downloaded_at], {:name=>"index_2669d36368", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0012s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_06", [:group_id, :iid, :downloaded_at], {:name=>"index_2669d36368", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0022s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_07", [:group_id, :iid, :downloaded_at], {:name=>"index_c13656fec7", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0013s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_07", [:group_id, :iid, :downloaded_at], {:name=>"index_c13656fec7", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0020s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_08", [:group_id, :iid, :downloaded_at], {:name=>"index_527048280a", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0013s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_08", [:group_id, :iid, :downloaded_at], {:name=>"index_527048280a", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0023s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_09", [:group_id, :iid, :downloaded_at], {:name=>"index_a587887746", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0012s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_09", [:group_id, :iid, :downloaded_at], {:name=>"index_a587887746", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0023s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_10", [:group_id, :iid, :downloaded_at], {:name=>"index_9c5f8f615f", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0015s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_10", [:group_id, :iid, :downloaded_at], {:name=>"index_9c5f8f615f", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0023s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_11", [:group_id, :iid, :downloaded_at], {:name=>"index_f32ef53c6f", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0012s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_11", [:group_id, :iid, :downloaded_at], {:name=>"index_f32ef53c6f", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0020s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_12", [:group_id, :iid, :downloaded_at], {:name=>"index_a5701f6698", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0013s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_12", [:group_id, :iid, :downloaded_at], {:name=>"index_a5701f6698", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0022s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_13", [:group_id, :iid, :downloaded_at], {:name=>"index_3954ae28dd", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0012s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_13", [:group_id, :iid, :downloaded_at], {:name=>"index_3954ae28dd", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0023s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_14", [:group_id, :iid, :downloaded_at], {:name=>"index_2b154c3219", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0013s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_14", [:group_id, :iid, :downloaded_at], {:name=>"index_2b154c3219", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0021s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- index_exists?("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_15", [:group_id, :iid, :downloaded_at], {:name=>"index_db5aa5426d", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0012s
ci: -- add_index("gitlab_partitions_static.virtual_registries_container_cache_remote_entries_15", [:group_id, :iid, :downloaded_at], {:name=>"index_db5aa5426d", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0027s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- add_index(:virtual_registries_container_cache_remote_entries, [:group_id, :iid, :downloaded_at], {:name=>:index_vr_cont_cache_rem_entries_on_requiring_cleanup, :where=>"status = 0"})
ci:    -> 0.0023s
ci: == 20260130154428 AddRequiringCleanupIdxToVirtualRegistriesContainerCacheRemEntries: migrated (0.1124s)

ci: == [advisory_lock_connection] object_id: 134060, pg_backend_pid: 90677

How to set up and validate locally

Automated tests should cover the use cases that the change relates to. The maintainer (domain expert from devopspackage) can also perform sanity checks that ensure that the changes are complete and accurate.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #583726 (closed)

Edited Jan 30, 2026 by Adie (she/her)

Use the new Cache::Remote::Entry model for container virtual registry