Skip to content

Add GraphQL support to manage the Packages Cleanup policy

David Fernandez requested to merge 346153-cleanup-policies-apis into master

Context

The Packages Registry works with these core models (simplified):

flowchart LR
    Group -- 1:n --> Project
    Project -- 1:n --> Package
    Package -- 1:n --> PackageFile
    PackageFile -- 1:1 --> os((Object Storage file))

For some package formats, we allow republishing a package version. What happens is that we append the package files to the existing package.

With time, some packages can end up with many package files. All these package files take space on Object Storage.

With #346153 (closed), we're kicking off the work for adding cleanup policies for packages. Basically, users will be allowed to define cleanup rules and the backend will regularly execute the policy to remove packages/package files that are not kept by the policy.

In true iteration spirit, the first iteration will have a single rule. Users will be able to define how many duplicated package files (by filename) need to be kept.

Example: for maven package, a pom.xml is uploaded on each publication. If you publish the same version 100 times, you end up with 100 pom.xml package files. Users will be able to state that they only want to keep the 10 most recent pom.xml files.

For this feature, there are 3 parts:

  1. The policy model. That's !85918 (merged).
  2. Expose the policy object through GraphQL 👈 This MR.
    • This is needed by the frontend to display the policy and its "rules".
  3. The background job that executes the cleanup policies.

As stated above, this MR introduces the GraphQL pieces to manage the packages cleanup policy.

Lastly, understand that we're setting up things for the first iteration of packages cleanup policies with only 1 rule. At some point down the road, we're going to expand these policies with more whistles and bells (eg. more rules that users can define).

What does this MR do and why?

  • Add a GraphQL Query to read the packages cleanup policy of a given project.
  • Add a GraphQL Mutation to create/update the packages cleanup policy for a given project.
  • Add a service to create/update a given policy. This service is used by the GraphQL Mutation.
  • Add all related specs.
  • Authorization wise, we're using the admin_package that we introduced in !86281 (merged). The difference here is that we need that permission in the Project policy.
    • We introduced this permission to gate all Package Registry settings operations. Since a Packages cleanup policy is considered a Package Registry setting, we simply used that same permission.
    • We're using the same rules here: only maintainer and owner will have this permission.

📺 Screenshots or screen recordings

  1. Let's see the policy of the first project:

    • GraphQL query
      {
        project(fullPath: "gitlab-org/gitlab-test") {
          id
          name
          packagesCleanupPolicy {
            keepNDuplicatedPackageFiles
            nextRunAt
          }
        }
      }
    • GraphQL response
      {
        "data": {
          "project": {
            "id": "gid://gitlab/Project/1",
            "name": "Gitlab Test",
            "packagesCleanupPolicy": {
              "keepNDuplicatedPackageFiles": "ALL_PACKAGE_FILES",
              "nextRunAt": null
            }
          }
        }
      }
    * We get a dummy policy with default values (disabled).
  2. Let's update the policy. We feel adventurous, so we want to keep only 1 package file when there are duplications:

    • GraphQL query
      mutation {
        updatePackagesCleanupPolicy(input: {
          projectPath: "gitlab-org/gitlab-test",
          keepNDuplicatedPackageFiles: ONE_PACKAGE_FILE
        }) {
          packagesCleanupPolicy {
            keepNDuplicatedPackageFiles
            nextRunAt
          }
        }
      }
    • GraphQL response
      {
        "data": {
          "updatePackagesCleanupPolicy": {
            "packagesCleanupPolicy": {
              "keepNDuplicatedPackageFiles": "ONE_PACKAGE_FILE",
              "nextRunAt": "2022-05-18T03:04:43Z"
            }
          }
        }
      }
    • The policy is created with the value we want

  3. Running the query again gives us:

    • GraphQL response
      {
        "data": {
          "project": {
            "id": "gid://gitlab/Project/1",
            "name": "Gitlab Test",
            "packagesCleanupPolicy": {
              "keepNDuplicatedPackageFiles": "ONE_PACKAGE_FILE",
              "nextRunAt": "2022-05-18T03:04:43Z"
            }
          }
        }
      }
    • All good

Now, let's see what happens with a developer member.

  1. Trying to read the policy with the query returns:
    • GraphQL response
      {
        "data": {
          "project": {
            "id": "gid://gitlab/Project/1",
            "name": "Gitlab Test",
            "packagesCleanupPolicy": null
          }
        }
      }
  2. Trying to update the policy with the mutation returns:
    • GraphQL response
      {
        "data": {
          "updatePackagesCleanupPolicy": null
        },
        "errors": [
          {
            "message": "The resource that you are attempting to access does not exist or you don't have permission to perform this action",
            "locations": [
              {
                "line": 2,
                "column": 3
              }
            ],
            "path": [
              "updatePackagesCleanupPolicy"
            ]
          }
        ]
      }
  3. Both access are denied

How to set up and validate locally

  1. Load the graphql editor: <gdk host>/-/graphql-explorer
  2. Run the queries the we used above.

🚓 MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Fernandez

Merge request reports