Catch and log error when failing to async delete container repositories

Context

Related to #375261 (closed). Found while investigating its root cause.

Problem

The DeleteContainerRepositoryWorker calls Projects::ContainerRepository::DestroyService to delete a container repository. There we do this:

container_repository.delete_tags!
container_repository.delete_failed! unless container_repository.destroy

If the call to delete_tags! fails for some reason (e.g., error deleting tag N out of M) we're not setting the repository status to delete_failed. We only flag it as delete_failed if failing to delete the corresponding ContainerRepository object (last step).

Proposal

Use Projects::ContainerRepository::DeleteTagsService to delete tags instead of calling ContainerRepository#delete_tags! as this service detects the proper way to delete tags. This service is also the one used by cleanup policies. See more information here: #375604 (comment 1118935553)

Using a feature flag to roll out the change is also good to lessen risk and see the effects gradually.

Edited by Adie (she/her)