Skip to content

Add jids redis key support to CleanupContainerRepositoryWorker

David Fernandez requested to merge 208193-add-cache-key-support-in-cleanup-container-repository-worker into master

🌲 Context

This is MR 2 of #208193 (comment 362910703) (see MR plan)

One upcoming feature is having Container Registry Cleanup Policies open to all projects (currently available only on projects from 12.7+). The policies are executed under a given schedule and each execution will use this worker that will call the proper service for tags deletion (which involves contacting the Container Registry for each tag removal). For a more detailed components view, see #208193 (comment 362910703) (see Current Situation)

This MR is part of a larger effort to limit resources usage during those Container Registry Cleanup Policies executions.

The main idea is that ContainerExpirationPolicyWorker will act as a monitor for CleanupContainerRepositoryWorker: at periodic intervals, it will enqueue CleanupContainerRepositoryWorker jobs up to a certain limit and will put their jids in Redis. During each invocation, CleanupContainerRepositoryWorker will then use this Redis entry to check the status of each job update the list for ongoing jobs and fill the "open slots" with more jobs. For all the details, see gitlab-com/gl-infra/scalability#461 (closed).

🤔 What does this MR do?

This MR prepares CleanupContainerRepositoryWorker for the above behavior in an upcoming MR. Basically, CleanupContainerRepositoryWorker will now:

  • accept a new optional parameter jids_redis_key, which is the Redis key under which ContainerExpirationPolicyWorker will put all the jids.
  • remove its own jid from the given jids_redis_key.
  • do the jid removal only under two conditions:
    • feature flag container_registry_expiration_policies_throttling is enabled. This feature flags controls all the throttling changes around expiration policies. See the tracking issue: #238190 (closed)
    • jids_cache_key has a non blank value
  • be idempotent. We can safely run the job with the exact same parameters multiple times.
  • not allow future duplicated executions. By future duplicated executions, we mean that we will reject two future executions with the exact same parameters. This is not useful, the second one will be a no op, so we might as well not enqueue a job for it.

Screenshots

n / a

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by David Fernandez

Merge request reports