Limit the number of container tags to delete in a single pass

🍍 Context

The CleanupTagsService is responsible for:

1. build the list of container registry tags to delete out of the passed container repository and parameters (A)
2. execute the actual delete looping on the list (B)

This service is mainly used by cleanup policies but it it can be invoked by the API too.

In !40740 (merged), we added several limits around (B) for "heavy repositories" (with a few thousands of tags). Those limits allow us to "stop" the actual delete and resume it at a later time when resources are available. This is done by using the limited capacity worker concern

With #288812 (closed), we observed that (A) has also issues with those heavy repositories. Let's detail how this list is built:

1. Get all the tags of the given container repository (⚡)
2. Exclude latest tags
3. Apply the regexs from the params on each tag name
4. Order the list by creation_date (⚡ per tag)
5. Keep the n most recent ones. n is param.
6. Exclude those more recent than older_than (from params)

⚡ denotes a network request to the container registry.

Obviously step (4.) is not scalable at all. Imagine having a container repository with 70K tags = 70K requests.

👁‍🗨 What does this MR do?

As explained in #288812 (closed), we will update the steps with:

1. Get all the tags of the given container repository (⚡)
2. Exclude latest tags
3. Apply the regexs from the params on each tag name
4. Truncate the list of tags if feature flag + application setting is set.
5. Order the list by creation_date (⚡ per tag)
6. Keep the n most recent ones. n is param.
7. Exclude those more recent than older_than (from params)

The new step (4.) will simply limit the total number of tags we can delete. This truncation will happen right before contacting the container registry for each creation_date.

This change is gated behind a feature flag : container_registry_expiration_policies_throttling. This is the feature flag we use to enable all the limits around deleting container registry tags.

The size of the list we extract in step (4.) is set by an application setting: container_registry_cleanup_tags_service_max_list_size.

This application setting is not available in the UI since it's behind a feature flag. Having said that, I plan to add them to the UI in an upcoming MR and properly update the documentation on how to use them, so that self-managed can try those limits and play with them.

The API documentation has been updated to add a note about this possible truncation of the list of tags to delete.

🔧 Caveat

With the approach above we're truncating the initial list of tags to delete in smaller lists and then apply filters (n most recent ones + older_than) on the small list. By doing so, we will inevitably exclude tags that should be deleted instead.

In short, with the above approach, we can exclude tags because they are excluded in the smaller list (they are excluded at the local scale) but on the complete list they wouldn't be excluded (they are not excluded at the global scale).

This will induce that we're making more network requests to the container registry by truncating the complete list. This is an acceptable tradeoff: more network request to process a part of a big list of tags to delete.

Notice that the deleting a tag is accurate while working on smaller lists meaning that a tag meant to be deleted in the smaller list would have been deleted too in the system was able to process the bigger list.

📷 Screenshots (strongly suggested)

Given this setup:

• container_registry_cleanup_tags_service_max_list_size: 5 (we're going to delete 5 tags max per pass)
• container_registry_expiration_policies_worker_capacity: 2 (only two active workers will work through the backlog)

Here is the target project with some container repositories and tags:

The policy has these attributes:

• delete everything
• keep the most recent tag

So we have a policy that keeps the most recent tag. For a given container repository with 20 tags, we will have these rounds:

1. truncate the list of 20 to 6 tags (max list size + keep_n), delete 5
2. truncate the list of 15 to 6 tags, delete 5
3. truncate the list of 10 to 6 tags, delete 5
4. don't truncate the list of 5 and delete 4

4 rounds for each container repository(x5) = we should have 20 jobs in total.

Between these rounds, the container repository must be marked as cleanup_unfinished because the list of tags to delete has been truncated.

Here is the background worker log (truncated + simplified for clarity)

{"severity":"INFO","time":"2021-01-08T15:23:22.751Z","cleanup_scheduled_count":5,"cleanup_unfinished_count":0,"cleanup_total_count":5,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:22.757Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"4ef0dc61ef6e27a26bf88f25","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:22.757Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"9bf8c129b633d103a9ae7acd","job_status":"start"}

{"severity":"INFO","time":"2021-01-08T15:23:23.413Z","cleanup_scheduled_count":3,"cleanup_unfinished_count":1,"cleanup_total_count":4,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:23.415Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"dea3f5efad564cd27c26acb5",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:23.422Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"4ef0dc61ef6e27a26bf88f25","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":116,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:23.477Z","cleanup_scheduled_count":3,"cleanup_unfinished_count":2,"cleanup_total_count":5,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:23.479Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"9bf8c129b633d103a9ae7acd","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":115,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}
{"severity":"INFO","time":"2021-01-08T15:23:23.480Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"5012d656cd439d06d38b7939",start","job_status":"start"}

{"severity":"INFO","time":"2021-01-08T15:23:24.061Z","cleanup_scheduled_count":1,"cleanup_unfinished_count":3,"cleanup_total_count":4,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:24.063Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"9446d39024bf233c86021e4d",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:24.064Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"dea3f5efad564cd27c26acb5","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":114,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:24.130Z","cleanup_scheduled_count":1,"cleanup_unfinished_count":4,"cleanup_total_count":5,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:24.134Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"8c193b031cdbc1b03dd7c637",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:24.135Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"5012d656cd439d06d38b7939","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":113,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:24.900Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":4,"cleanup_total_count":4,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:24.903Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"9446d39024bf233c86021e4d","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":112,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":20,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}
{"severity":"INFO","time":"2021-01-08T15:23:24.904Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"6571362ca2ff5330350292a4",start","job_status":"start"}

{"severity":"INFO","time":"2021-01-08T15:23:24.968Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":5,"cleanup_total_count":5,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:24.970Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"8c193b031cdbc1b03dd7c637","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":116,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}
{"severity":"INFO","time":"2021-01-08T15:23:24.971Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"59026cff8f3db4155a0c009c",start","job_status":"start"}

{"severity":"INFO","time":"2021-01-08T15:23:25.571Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":4,"cleanup_total_count":4,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:25.573Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"70c785f2ccfbe056ff7c496c",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:25.574Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"6571362ca2ff5330350292a4","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":115,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:25.639Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":5,"cleanup_total_count":5,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:25.640Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"8e610787039b181a226b7aa4",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:25.641Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"59026cff8f3db4155a0c009c","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":114,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:26.222Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":4,"cleanup_total_count":4,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:26.224Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"74655915fa4025e14bda3d68",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:26.225Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"70c785f2ccfbe056ff7c496c","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":113,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:26.285Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":5,"cleanup_total_count":5,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:26.287Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"8e610787039b181a226b7aa4","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":112,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":15,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}
{"severity":"INFO","time":"2021-01-08T15:23:26.288Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"9d405097ae98fcfd2b929bf2",start","job_status":"start"}

{"severity":"INFO","time":"2021-01-08T15:23:26.955Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":4,"cleanup_total_count":4,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:26.957Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"74655915fa4025e14bda3d68","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":116,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}
{"severity":"INFO","time":"2021-01-08T15:23:26.958Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"ae22fe66cdb8a35910c0f25f",start","job_status":"start"}

{"severity":"INFO","time":"2021-01-08T15:23:27.000Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":5,"cleanup_total_count":5,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:27.002Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"34652bb3a44be4ac41ef0b63",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:27.003Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"9d405097ae98fcfd2b929bf2","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":115,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:27.604Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":4,"cleanup_total_count":4,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:27.606Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"a4f3e7e2cec27fda5093553e",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:27.607Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"ae22fe66cdb8a35910c0f25f","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":114,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:27.673Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":5,"cleanup_total_count":5,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:27.675Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"af4af95114d182331d788c83",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:27.676Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"34652bb3a44be4ac41ef0b63","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":113,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:28.268Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":4,"cleanup_total_count":4,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:28.274Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"d126c0ed31f2dbc2bdc2d642",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:28.275Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"a4f3e7e2cec27fda5093553e","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":112,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"unfinished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":10,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":6,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":5}

{"severity":"INFO","time":"2021-01-08T15:23:28.284Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":4,"cleanup_total_count":4,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:28.286Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"e47f15e5eec79006ea18bf5e",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:28.287Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"af4af95114d182331d788c83","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":116,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"finished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":4}

{"severity":"INFO","time":"2021-01-08T15:23:28.906Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":2,"cleanup_total_count":2,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:28.908Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"2a523a5f3792445d60a99f9d",start","job_status":"start"}
{"severity":"INFO","time":"2021-01-08T15:23:28.910Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"d126c0ed31f2dbc2bdc2d642","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":115,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"finished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":4}

{"severity":"INFO","time":"2021-01-08T15:23:28.972Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":2,"cleanup_total_count":2,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:28.974Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"e47f15e5eec79006ea18bf5e","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":114,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"finished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":4}
{"severity":"INFO","time":"2021-01-08T15:23:28.975Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"46f17ce76299ec01412c34c9",start","job_status":"start"}

{"severity":"INFO","time":"2021-01-08T15:23:29.496Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":0,"cleanup_total_count":0,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:29.497Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"2a523a5f3792445d60a99f9d","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":113,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"finished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":4}

{"severity":"INFO","time":"2021-01-08T15:23:29.558Z","cleanup_scheduled_count":0,"cleanup_unfinished_count":0,"cleanup_total_count":0,"job_status":"running"}

{"severity":"INFO","time":"2021-01-08T15:23:29.559Z","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","jid":"46f17ce76299ec01412c34c9","job_status":"done","extra.container_expiration_policies_cleanup_container_repository_worker.container_repository_id":112,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_status":"finished","extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_original_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_after_truncate_size":5,"extra.container_expiration_policies_cleanup_container_repository_worker.cleanup_tags_service_before_delete_size":4}

-> we have 20 "start" job events. ✅

I recall that we have a minor bug on the limited capacity worker that could enqueue slightly more jobs than the limited capacity. This would explain the extra "start" job event.

• We clearly see the counts of cleanup_scheduled going from 5 to 0. At the same time, the cleanup_unfinished count goes from 0 to 5.
  • cleanup_scheduled have the highest priority.
  • container repositories are properly marked as cleanup_unfinished.
• It stays a while on cleanup_unfinished 4 and 5. That's all "intermediary" rounds.
• We can then observe that the cleanup_unfinished goes from 5 to 0.
• The done messages have the counts for the truncation operation: we can see that the list is truncated to 6 and 5 tags are deleted (because keep_n = 1)

Here is the result on the UI after all this cleanup:

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• [-] Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• [-] Label as security and @ mention @gitlab-com/gl-security/appsec
• [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• [-] Security reports checked/validated by a reviewer from the AppSec team

Database review

Migration Up

== 20201214111858 AddContainerRegistryCleanupTagsServiceMaxListSizeToApplicationSettings: migrating 
-- column_exists?(:application_settings, :container_registry_cleanup_tags_service_max_list_size)
   -> 0.0342s
-- add_column(:application_settings, :container_registry_cleanup_tags_service_max_list_size, :integer, {:default=>200, :null=>false})
   -> 0.0088s
== 20201214111858 AddContainerRegistryCleanupTagsServiceMaxListSizeToApplicationSettings: migrated (0.0432s)

== 20201214112752 AddAppSettingsContainerRegCleanupTagsServiceMaxListSizeConstraint: migrating 
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE application_settings\nADD CONSTRAINT app_settings_container_reg_cleanup_tags_max_list_size_positive\nCHECK ( container_registry_cleanup_tags_service_max_list_size >= 0 )\nNOT VALID;\n")
   -> 0.0017s
-- current_schema()
   -> 0.0001s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT app_settings_container_reg_cleanup_tags_max_list_size_positive;")
   -> 0.0007s
-- execute("RESET ALL")
   -> 0.0002s
== 20201214112752 AddAppSettingsContainerRegCleanupTagsServiceMaxListSizeConstraint: migrated (0.0083s) 

Migration down

== 20201214112752 AddAppSettingsContainerRegCleanupTagsServiceMaxListSizeConstraint: reverting 
-- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS app_settings_container_reg_cleanup_tags_max_list_size_positive\n")
   -> 0.0013s
== 20201214112752 AddAppSettingsContainerRegCleanupTagsServiceMaxListSizeConstraint: reverted (0.0065s) 

== 20201214111858 AddContainerRegistryCleanupTagsServiceMaxListSizeToApplicationSettings: reverting 
-- column_exists?(:application_settings, :container_registry_cleanup_tags_service_max_list_size)
   -> 0.0349s
-- remove_column(:application_settings, :container_registry_cleanup_tags_service_max_list_size)
   -> 0.0017s
== 20201214111858 AddContainerRegistryCleanupTagsServiceMaxListSizeToApplicationSettings: reverted (0.0367s)