A very large number of noop CleanupContainerRepositoryWorker runs for a single project
Summary
GitLab team members can read more in the ticket.
- Customer observed constant large volumes of S3 requests from GitLab
- Root cause was identified as CleanupContainerRepositoryWorker
- 07:29 to 07:33: 1912 log entries in sidekiq for
"class": "ContainerExpirationPolicies::CleanupContainerRepositoryWorker"
withjob_status: done
. - All for one project.
- Two
container_repository_id
- Original tag size ~800 for both
-
after_truncate_size
250 -
deleted_size
always zero
- 07:29 to 07:33: 1912 log entries in sidekiq for
Project image tag cleanup settings:
- enabled
- weekly
- next scheduled run is 24 hours from now, and the logs we have are for yesterday, so this suggests the cleanup from last week is still running
- keep 50 tags per image name
- rule: remove tags older than 90 days, matching
.*
Steps to reproduce
See: #362914 (comment 953530856)
Example Project
What is the current bug behavior?
- Vast number of
ContainerExpirationPolicies::CleanupContainerRepositoryWorker
instances being started.
What is the expected correct behavior?
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
14.10.0
Possible fixes
Edited by Ben Prescott_