Container Expiration Policies scheduled deletion does not work
Summary
- Container Registry tag expiration policy background cron job triggered just fine.
- However, when the tags expired, the background jobs triggered did not delete the tags.
- This is observed on the GitLab 12.10.x & 13.0.x
- Other versions may be affected but have not been tested
Steps to reproduce
- Inspect ContainerExpirationPolicy of the project
- ContainerExpirationPolicy.find_by(project_id: $project_id)
- Update
next_run_at
to sometime in the past- This will trigger the ContainerExpirationPolicyService to execute
- You'll probably need to update the
next_run_at
directly on the database so that you can trigger this immediately
- Make sure you have tags that are supposed to be expired already as per the tags expiration policy
- Force the cron job to run immediately
-
gitlab.url/admin/background_jobs
>cron
>container_expiration_policy_worker
> click on Enqueue now
-
- Expired tags are not removed
What is the current bug behavior?
- Tags expired but not removed from the container registry.
What is the expected correct behavior?
- Expired tags removed as per the tags expiration policy.
Possible fixes
- The bug seems to be introduced on this MR that fixes another bug with Container Expiration Policy
- Project owner user is pass through from ContainerExpirationPolicyWorker
-
nil
user ID is passed through from ContainerExpirationPolicyService - CleanupTagsService still run because of the params['container_expiration_policy'] is set to true
-
Similar override is not present on DeleteTagsService
- This means that when
current_user
isnil
, DeleteTagsService never runs.
- This means that when
Note the open issue that might tough this: User auditing for container expiration and retention policies
Possible fix:
- Add a similar override on DeleteTagsService
- Pass a user ID through
Workaround
To trigger the deletion of expired tags, run the following from console. This passes through a user that has permission to delete the tags & will delete all the tags as per the container expiration policy,
### Find a user that has permission to delete tags/images
u = User.find_by_id(USER_ID)
### Get the details of the project/container registry & expiration policy:
p = Project.find_by_id(PROJECT_ID)
cr = ContainerRepository.find_by(project_id: PROJECT_ID)
cep =ContainerExpirationPolicy.find_by(project_id: PROJECT_ID)
Projects::ContainerRepository::CleanupTagsService.new(p, u, cep.attributes.except("created_at", "updated_at")).execute(cr)
Reference
ZD internal-only customer report: https://gitlab.zendesk.com/agent/tickets/159299
Edited by Andrew Winata