Maven VReg: Background job to destroy stale cached responses
Context
Follow up of Maven Registry: Cached responses destruction (#468113 - closed).
In Maven Registry: Cached responses destruction (#468113 - closed), we converted upstream_id
column in the virtual_registries_packages_maven_cached_responses
database table into a loose foreign key with on_delete: async_nullify
action. That means when an upstream got deleted, its children in the virtual_registries_packages_maven_cached_responses
database table will be updated to have their upstream_id
column as null
.
Those stale/orphan rows should be deleted too. However, they are associated with external files (files stored on the Object Storage), that's why their destruction should be carried out by a background job (cleaner).
What does this MR do and why?
- Add a new status database column to the
virtual_registries_packages_maven_cached_responses
table to be used in the destruction logic. - Add a cron job that will detect
pending_destruction
records. If there is any work, enqueue a limited capacity job. - Add a limited capacity job that will be responsible for destroying the stale records, slowly one by one.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Make sure you have rails-background-jobs up & running in your GDK
- Start rails console
# stub file upload
def fixture_file_upload(*args, **kwargs)
Rack::Test::UploadedFile.new(*args, **kwargs)
end
# create two cached responses that belong to the upstream
cr = FactoryBot.create(:virtual_registries_packages_maven_cached_response)
stale_crs = FactoryBot.create_list(:virtual_registries_packages_maven_cached_response, 3, :orphan)
# run the cleanup worker
DependencyProxy::CleanupDependencyProxyWorker.new.perform
# cr wasn't deleted because it's not stale
VirtualRegistries::Packages::Maven::CachedResponse.find(cr.id)
=> #<VirtualRegistries::Packages::Maven::CachedResponse ...
# stale_crs are deleted
VirtualRegistries::Packages::Maven::CachedResponse.pending_destruction.size
=> 0
Related to #479957