Skip to content

Maven VReg: Background job to destroy stale cached responses

Moaz Khalifa requested to merge 479957-maven-vreg-add-worker-to-destroy-stale-cached-responses into master

Context

Follow up of Maven Registry: Cached responses destruction (#468113 - closed).

In Maven Registry: Cached responses destruction (#468113 - closed), we converted upstream_id column in the virtual_registries_packages_maven_cached_responses database table into a loose foreign key with on_delete: async_nullify action. That means when an upstream got deleted, its children in the virtual_registries_packages_maven_cached_responses database table will be updated to have their upstream_id column as null.

Those stale/orphan rows should be deleted too. However, they are associated with external files (files stored on the Object Storage), that's why their destruction should be carried out by a background job (cleaner).

What does this MR do and why?

  • Add a new status database column to the virtual_registries_packages_maven_cached_responses table to be used in the destruction logic.
  • Add a cron job that will detect pending_destruction records. If there is any work, enqueue a limited capacity job.
  • Add a limited capacity job that will be responsible for destroying the stale records, slowly one by one.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Make sure you have rails-background-jobs up & running in your GDK
  2. Start rails console
# stub file upload
def fixture_file_upload(*args, **kwargs)
  Rack::Test::UploadedFile.new(*args, **kwargs)
end

# create two cached responses that belong to the upstream
cr = FactoryBot.create(:virtual_registries_packages_maven_cached_response)
stale_crs = FactoryBot.create_list(:virtual_registries_packages_maven_cached_response, 3, :orphan)

# run the cleanup worker 
DependencyProxy::CleanupDependencyProxyWorker.new.perform

# cr wasn't deleted because it's not stale
 VirtualRegistries::Packages::Maven::CachedResponse.find(cr.id)
 => #<VirtualRegistries::Packages::Maven::CachedResponse ...  

 # stale_crs are deleted
 VirtualRegistries::Packages::Maven::CachedResponse.pending_destruction.size
 => 0

Related to #479957

Edited by David Fernandez

Merge request reports