Set up partitioning for maven virtual registry cached responses

⚖️ Context

In Maven virtual registry MVC (API only interactions) (&14137 - closed), we're working for the first version of Maven virtual registries.

The model that we used is the following:

Group -1:1-> Registry <-1:1- RegistryUpstream -1:1-> Upstream -1:n-> CachedResponses

CachedResponses are responses from Upstream that we store on object storage. As such, this table contains a reference or better said a key to a file on object storage.

During the creation of the table of the CachedResponse, it was asked if we needed partitioning for that table.

In #473144 (comment 2244331754), given the expected conditions, partitioning could help with the main query which is reading a record out of that table.

Now, a record has essentially a few fields related to the response that is stored. The most important ones:

• upstream_id, the foreign key to the parent Upstream.
• group_id, this foreign key is used as the mandatory sharding key.
• status. Given that we have a reference to object storage, destroying a cached response involves also contacting object storage to delete the file there. Thus, we have a system where we "mark" records as pending destruction, then we have a cleanup job that will slowly destroy those records and the files on object storage. This is to avoid having to deal with a 🌊 of DELETE which in turn would mean an 🌊 of network calls to object storage. With this system, a 🌊 of DELETE = an 🌊 of UPDATE.
• relative_path. This is the relative path that uniquely identifies the file. Maven package registries organize packages and files in a file like system (with a hierarchy of folders). Thus, a relative_path is like an identifier to locate a given file.

We also have a few constraints on the table. The main one is that, at any given time, only 1 record can have the same upstream_id and relative_path for records that are in the :default status.

Given the above, this MR tries to migrate the table to a partitioned table.

🤔 What does this MR do and why?

• Introduce a new table: virtual_registries_packages_maven_cache_entries which is the partitioned version of virtual_registries_packages_maven_cached_responses table (16 partitions).
  • In Finalize virtual_registries_packages_maven_cach... (#512436 - closed) • David Fernandez • 17.9 • On track, we're going to rename the models properly and drop the old table.
• Update the model to connect the new table.
• Update different backend parts. See comments.
  • We took this opportunity to fine tune some validations. For example, file_sha1 and file_md5 columns will no expect values with an exact length. Anything else will be rejected.
• Functionally wise, there is non change.

The maven virtual registry is behind a WIP feature flag. As such, switching the existing model to a new table is fine. On gitlab.com, the feature is only used by 2 team members and is currently not used (virtual_registries_packages_maven_cached_responses is empty). On self-managed, even though it's a WIP feature flag, we could make records of virtual_registries_packages_maven_cached_responses not reachable anymore. That's ok. The table represents a cache. Thus, functionally speaking, nothing will be broken. At worst, the entries will be "re created" in the new table when necessary.

📚 References

Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

• #473144 (closed)
• https://docs.gitlab.com/ee/development/database/partitioning/hash.html

⚙️ MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

✅

🌈 Screenshots or screen recordings

No UI changes

⚙️ How to set up and validate locally

Follow !173687 (merged), that will create a cached response record.

Then in the rails console, you can play around with:

• destroying the Upstream.
• destroying the Group.
• using the CachedResponse API to search through them.

💾 Database review

⤴️ Migration up

Migration up

$ r db:migrate:up:main VERSION=20241224105102 && r db:migrate:up:ci VERSION=20241224105102
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/david/projects/gitlab-development-kit/gitlab/config/environment.rb:7)
main: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 51601
main: == 20241224105102 CreateVirtualRegistriesPackagesMavenCacheEntries: migrating =
main: -- create_table(:virtual_registries_packages_maven_cache_entries, {:if_not_exists=>true, :options=>"PARTITION BY HASH (relative_path)", :primary_key=>[:upstream_id, :relative_path, :status]})
main: -- quote_column_name(:relative_path)
main:    -> 0.0000s
main: -- quote_column_name(:file)
main:    -> 0.0000s
main: -- quote_column_name(:object_storage_key)
main:    -> 0.0000s
main: -- quote_column_name(:upstream_etag)
main:    -> 0.0000s
main: -- quote_column_name(:content_type)
main:    -> 0.0000s
main: -- quote_column_name(:file_final_path)
main:    -> 0.0000s
main:    -> 0.0173s
main: -- transaction(nil)
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_00\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 0);\n")
main:    -> 0.0189s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_01\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 1);\n")
main:    -> 0.0070s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_02\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 2);\n")
main:    -> 0.0040s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_03\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 3);\n")
main:    -> 0.0038s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_04\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 4);\n")
main:    -> 0.0052s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_05\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 5);\n")
main:    -> 0.0045s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_06\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 6);\n")
main:    -> 0.0054s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_07\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 7);\n")
main:    -> 0.0053s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_08\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 8);\n")
main:    -> 0.0044s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_09\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 9);\n")
main:    -> 0.0054s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_10\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 10);\n")
main:    -> 0.0040s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_11\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 11);\n")
main:    -> 0.0043s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_12\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 12);\n")
main:    -> 0.0043s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_13\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 13);\n")
main:    -> 0.0063s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_14\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 14);\n")
main:    -> 0.0044s
main: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_15\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 15);\n")
main:    -> 0.0057s
main:    -> 0.0933s
main: == 20241224105102 CreateVirtualRegistriesPackagesMavenCacheEntries: migrated (0.1463s) 

main: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 51601
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/david/projects/gitlab-development-kit/gitlab/config/environment.rb:7)
ci: == [advisory_lock_connection] object_id: 131140, pg_backend_pid: 52010
ci: == 20241224105102 CreateVirtualRegistriesPackagesMavenCacheEntries: migrating =
ci: -- create_table(:virtual_registries_packages_maven_cache_entries, {:if_not_exists=>true, :options=>"PARTITION BY HASH (relative_path)", :primary_key=>[:upstream_id, :relative_path, :status]})
ci: -- quote_column_name(:relative_path)
ci:    -> 0.0000s
ci: -- quote_column_name(:file)
ci:    -> 0.0000s
ci: -- quote_column_name(:object_storage_key)
ci:    -> 0.0000s
ci: -- quote_column_name(:upstream_etag)
ci:    -> 0.0000s
ci: -- quote_column_name(:content_type)
ci:    -> 0.0000s
ci: -- quote_column_name(:file_final_path)
ci:    -> 0.0000s
ci:    -> 0.0155s
ci: -- transaction(nil)
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_00\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 0);\n")
ci:    -> 0.0156s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_01\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 1);\n")
ci:    -> 0.0048s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_02\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 2);\n")
ci:    -> 0.0045s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_03\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 3);\n")
ci:    -> 0.0038s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_04\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 4);\n")
ci:    -> 0.0048s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_05\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 5);\n")
ci:    -> 0.0040s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_06\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 6);\n")
ci:    -> 0.0046s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_07\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 7);\n")
ci:    -> 0.0037s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_08\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 8);\n")
ci:    -> 0.0044s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_09\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 9);\n")
ci:    -> 0.0047s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_10\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 10);\n")
ci:    -> 0.0044s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_11\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 11);\n")
ci:    -> 0.0037s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_12\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 12);\n")
ci:    -> 0.0041s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_13\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 13);\n")
ci:    -> 0.0034s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_14\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 14);\n")
ci:    -> 0.0042s
ci: -- execute("CREATE TABLE gitlab_partitions_static.virtual_registries_packages_maven_cache_entries_15\nPARTITION OF virtual_registries_packages_maven_cache_entries\nFOR VALUES WITH (MODULUS 16, REMAINDER 15);\n")
ci:    -> 0.0051s
ci:    -> 0.0803s
I, [2025-01-10T10:51:58.916283 #51855]  INFO -- : Database: 'ci', Table: 'virtual_registries_packages_maven_cache_entries': Lock Writes
ci: == 20241224105102 CreateVirtualRegistriesPackagesMavenCacheEntries: migrated (0.1581s) 

ci: == [advisory_lock_connection] object_id: 131140, pg_backend_pid: 52010

r db:migrate:up:main VERSION=20250110090206 && r db:migrate:up:ci VERSION=20250110090206
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/david/projects/gitlab-development-kit/gitlab/config/environment.rb:7)
main: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 52428
main: == 20250110090206 AddCheckConstraintsToVirtualRegistriesPackagesMavenCacheEntries: migrating 
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- execute("ALTER TABLE virtual_registries_packages_maven_cache_entries\nADD CONSTRAINT check_cc222855d6\nCHECK ( file_md5 IS NULL OR octet_length(file_md5) = 16 )\nNOT VALID;\n")
main:    -> 0.0035s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- execute("ALTER TABLE virtual_registries_packages_maven_cache_entries VALIDATE CONSTRAINT check_cc222855d6;")
main:    -> 0.0039s
main: -- execute("RESET statement_timeout")
main:    -> 0.0005s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- execute("ALTER TABLE virtual_registries_packages_maven_cache_entries\nADD CONSTRAINT check_f2ea43b900\nCHECK ( octet_length(file_sha1) = 20 )\nNOT VALID;\n")
main:    -> 0.0015s
main: -- execute("ALTER TABLE virtual_registries_packages_maven_cache_entries VALIDATE CONSTRAINT check_f2ea43b900;")
main:    -> 0.0011s
main: == 20250110090206 AddCheckConstraintsToVirtualRegistriesPackagesMavenCacheEntries: migrated (0.0691s) 

main: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 52428
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/david/projects/gitlab-development-kit/gitlab/config/environment.rb:7)
ci: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 52836
ci: == 20250110090206 AddCheckConstraintsToVirtualRegistriesPackagesMavenCacheEntries: migrating 
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- execute("ALTER TABLE virtual_registries_packages_maven_cache_entries\nADD CONSTRAINT check_cc222855d6\nCHECK ( file_md5 IS NULL OR octet_length(file_md5) = 16 )\nNOT VALID;\n")
ci:    -> 0.0053s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0003s
ci: -- execute("ALTER TABLE virtual_registries_packages_maven_cache_entries VALIDATE CONSTRAINT check_cc222855d6;")
ci:    -> 0.0048s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0004s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- execute("ALTER TABLE virtual_registries_packages_maven_cache_entries\nADD CONSTRAINT check_f2ea43b900\nCHECK ( octet_length(file_sha1) = 20 )\nNOT VALID;\n")
ci:    -> 0.0021s
ci: -- execute("ALTER TABLE virtual_registries_packages_maven_cache_entries VALIDATE CONSTRAINT check_f2ea43b900;")
ci:    -> 0.0015s
ci: == 20250110090206 AddCheckConstraintsToVirtualRegistriesPackagesMavenCacheEntries: migrated (0.0884s) 

ci: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 52836

⤵️ Migration down

Migration down

r db:migrate:down:main VERSION=20250110090206 && r db:migrate:down:ci VERSION=20250110090206
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/david/projects/gitlab-development-kit/gitlab/config/environment.rb:7)
main: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 49926
main: == 20250110090206 AddCheckConstraintsToVirtualRegistriesPackagesMavenCacheEntries: reverting 
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- execute("            ALTER TABLE virtual_registries_packages_maven_cache_entries\n            DROP CONSTRAINT IF EXISTS check_cc222855d6\n")
main:    -> 0.0107s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- execute("            ALTER TABLE virtual_registries_packages_maven_cache_entries\n            DROP CONSTRAINT IF EXISTS check_f2ea43b900\n")
main:    -> 0.0003s
main: == 20250110090206 AddCheckConstraintsToVirtualRegistriesPackagesMavenCacheEntries: reverted (0.0315s) 

main: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 49926
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/david/projects/gitlab-development-kit/gitlab/config/environment.rb:7)
ci: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 50331
ci: == 20250110090206 AddCheckConstraintsToVirtualRegistriesPackagesMavenCacheEntries: reverting 
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- execute("            ALTER TABLE virtual_registries_packages_maven_cache_entries\n            DROP CONSTRAINT IF EXISTS check_cc222855d6\n")
ci:    -> 0.0011s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- execute("            ALTER TABLE virtual_registries_packages_maven_cache_entries\n            DROP CONSTRAINT IF EXISTS check_f2ea43b900\n")
ci:    -> 0.0004s
ci: == 20250110090206 AddCheckConstraintsToVirtualRegistriesPackagesMavenCacheEntries: reverted (0.0442s) 

ci: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 50331

r db:migrate:down:main VERSION=20241224105102 && r db:migrate:down:ci VERSION=20241224105102

main: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 84998
main: == 20241224105102 CreateVirtualRegistriesPackagesMavenCacheEntries: reverting =
main: -- drop_table(:virtual_registries_packages_maven_cache_entries)
main:    -> 0.0290s
main: == 20241224105102 CreateVirtualRegistriesPackagesMavenCacheEntries: reverted (0.0322s) 

main: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 84998
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/david/projects/gitlab-development-kit/gitlab/config/environment.rb:7)
ci: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 85418
ci: == 20241224105102 CreateVirtualRegistriesPackagesMavenCacheEntries: reverting =
ci: -- drop_table(:virtual_registries_packages_maven_cache_entries)
ci:    -> 0.0393s
ci: == 20241224105102 CreateVirtualRegistriesPackagesMavenCacheEntries: reverted (0.0475s) 

ci: == [advisory_lock_connection] object_id: 131040, pg_backend_pid: 85418

🔬 Queries analysis

Query	Analysis
Main read query (getting a cached response)	!174985 (comment 2250404167)
Find or initialize by (main query + group_id)	!174985 (comment 2250440117)
List cached responses of an upstream (without search param)	!174985 (comment 2250482980)
List cached responses of an upstream (with search param)	!174985 (comment 2250507340)
Cleanup job: pending destruction cached response presence check	!174985 (comment 2250554792)
Cleanup job: pending destruction cached response (limited) count	!174985 (comment 2250786876)
Cleanup job: get first pending destruction cached response	!174985 (comment 2250811921)
LFK update column to: parent upstream destroyed	!174985 (comment 2250916800)
LFK update column to: parent group destroyed	!174985 (comment 2250958271)