Container Expiration Policies scheduled deletion does not work

Summary

• Container Registry tag expiration policy background cron job triggered just fine.
• However, when the tags expired, the background jobs triggered did not delete the tags.
• This is observed on the GitLab 12.10.x & 13.0.x
  • Other versions may be affected but have not been tested

Steps to reproduce

• Inspect ContainerExpirationPolicy of the project
  • ContainerExpirationPolicy.find_by(project_id: $project_id)
• Update next_run_at to sometime in the past
  • This will trigger the ContainerExpirationPolicyService to execute
  • You'll probably need to update thenext_run_at directly on the database so that you can trigger this immediately
• Make sure you have tags that are supposed to be expired already as per the tags expiration policy
• Force the cron job to run immediately
  • gitlab.url/admin/background_jobs > cron > container_expiration_policy_worker > click on Enqueue now
• Expired tags are not removed

What is the current bug behavior?

• Tags expired but not removed from the container registry.

What is the expected correct behavior?

• Expired tags removed as per the tags expiration policy.

Possible fixes

• The bug seems to be introduced on this MR that fixes another bug with Container Expiration Policy
• Project owner user is pass through from ContainerExpirationPolicyWorker
• nil user ID is passed through from ContainerExpirationPolicyService
• CleanupTagsService still run because of the params['container_expiration_policy'] is set to true
• Similar override is not present on DeleteTagsService
  • This means that when current_user is nil, DeleteTagsService never runs.

Note the open issue that might tough this: User auditing for container expiration and retention policies

Possible fix:

1. Add a similar override on DeleteTagsService
2. Pass a user ID through

Workaround

To trigger the deletion of expired tags, run the following from console. This passes through a user that has permission to delete the tags & will delete all the tags as per the container expiration policy,

### Find a user that has permission to delete tags/images
u = User.find_by_id(USER_ID)

### Get the details of the project/container registry & expiration policy:
p = Project.find_by_id(PROJECT_ID)
cr = ContainerRepository.find_by(project_id: PROJECT_ID)
cep =ContainerExpirationPolicy.find_by(project_id: PROJECT_ID)

Projects::ContainerRepository::CleanupTagsService.new(p, u, cep.attributes.except("created_at", "updated_at")).execute(cr)

Reference

ZD internal-only customer report: https://gitlab.zendesk.com/agent/tickets/159299