Skip to content

Add the packages cleanup policy model

David Fernandez requested to merge 346153-cleanup-policy-model into master

🍋 Context

The Packages Registry works with these core models (simplified):

flowchart LR
    Group -- 1:n --> Project
    Project -- 1:n --> Package
    Package -- 1:n --> PackageFile
    PackageFile -- 1:1 --> os((Object Storage file))

For some package formats, we allow republishing a package version. What happens is that we append the package files to the existing package.

With time, some packages can end up with many package files. All these package files take space on Object Storage.

With #346153 (closed), we're kicking off the work for adding cleanup policies for packages. Basically, users will be allowed to define cleanup rules and the backend will regularly execute the policy to remove packages/package files that are not kept by the policy.

In true iteration spirit, the first iteration will have a single rule. Users will be able to define how many duplicated package files (by filename) need to be kept.

Example: for maven package, a pom.xml is uploaded on each publication. If you publish the same version 100 times, you end up with 100 pom.xml package files. Users will be able to state that they only want to keep the 10 most recent pom.xml files.

For this feature, there are 3 parts:

  1. The policy model 👈 This MR.
  2. Expose the policy object through GraphQL
    • This is needed by the frontend to display the policy and its "rules".
  3. The background job that executes the cleanup policies.

As stated above, this MR only introduces the cleanup policy model. The model is very simple for now but no doubt that (3.) will add a bunch of scopes to order and pick up the next policy to execute.

🔬 What does this MR do and why?

  • Add the Packages::Cleanup::Policy model.
  • Update Schedulable to not rely on an active scope.
    • This concern should only interact with the next_run_at field.
    • Reading or using a scope that is not defined in the concern makes it harder to re use. That's the case with Packages::Cleanup::Policy that doesn't have an active scope.
  • Update the related specs

🖥 Screenshots or screen recordings

n / a

How to set up and validate locally

Packages cleanup policies will not be created by default but only when needed. As such accessing a policy that doesn't exist yet on the database will build the object.

  1. In a rails console:
    Project.first.packages_cleanup_policy
    => #<Packages::Cleanup::Policy:0x00007ffc85e40b18 created_at: nil, updated_at: nil, project_id: 1, next_run_at: nil, keep_n_duplicated_package_files: "all">

🚥 MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Fernandez

Merge request reports