Infinite loop in ContainerRegistry::DeleteContainerRepositoryWorker
Summary
My gitlab instance is pegging the CPU. The logs show this over and over:
==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"INFO","time":"2024-05-03T05:10:52.094Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"b3cc39cfff4d3c2553fb73cea1680888","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","container_repository_id":1,"container_repository_path":"home/clarkweb","project_id":4,"third_party_cleanup_tags_service":true}
{"severity":"ERROR","time":"2024-05-03T05:10:52.151Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"b3cc39cfff4d3c2553fb73cea1680888","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","service_class":"Projects::ContainerRepository::DeleteTagsService","container_repository_id":1,"project_id":4,"message":"could not delete tags: latest"}
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2024-05-03T05:10:52.151Z","class":"ContainerRegistry::DeleteContainerRepositoryWorker","project_id":4,"container_repository_id":1,"container_repository_path":"home/clarkweb","tags_size_before_delete":1,"deleted_tags_size":null,"meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"b3cc39cfff4d3c2553fb73cea1680888","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","job_status":"running","queue":"default","jid":"c77069ad12c1b287315366ff","retry":0}
{"severity":"INFO","time":"2024-05-03T05:10:52.164Z","retry":0,"queue":"default","version":0,"store":null,"status_expiration":1800,"queue_namespace":"container_repository_delete","args":[],"class":"ContainerRegistry::DeleteContainerRepositoryWorker","jid":"c77069ad12c1b287315366ff","created_at":"2024-05-03T05:10:52.072Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"b3cc39cfff4d3c2553fb73cea1680888","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":"2024-05-03T05:10:52.073Z","job_size_bytes":2,"pid":503,"message":"ContainerRegistry::DeleteContainerRepositoryWorker JID-c77069ad12c1b287315366ff: done: 0.090109 sec","job_status":"done","scheduling_latency_s":0.001062,"redis_calls":9,"redis_duration_s":0.002142,"redis_read_bytes":10,"redis_write_bytes":1358,"redis_queues_calls":2,"redis_queues_duration_s":0.000264,"redis_queues_read_bytes":2,"redis_queues_write_bytes":614,"redis_queues_metadata_calls":2,"redis_queues_metadata_duration_s":0.000807,"redis_queues_metadata_read_bytes":3,"redis_queues_metadata_write_bytes":105,"redis_shared_state_calls":5,"redis_shared_state_duration_s":0.001071,"redis_shared_state_read_bytes":5,"redis_shared_state_write_bytes":639,"db_count":10,"db_write_count":3,"db_cached_count":1,"db_txn_count":1,"db_replica_txn_count":0,"db_primary_txn_count":0,"db_main_txn_count":1,"db_ci_txn_count":0,"db_main_replica_txn_count":0,"db_ci_replica_txn_count":0,"db_replica_count":0,"db_primary_count":10,"db_main_count":10,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_main_cached_count":1,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_txn_max_duration_s":0.0,"db_primary_txn_max_duration_s":0.0,"db_main_txn_max_duration_s":0.006,"db_ci_txn_max_duration_s":0.0,"db_main_replica_txn_max_duration_s":0.0,"db_ci_replica_txn_max_duration_s":0.0,"db_replica_txn_duration_s":0.0,"db_primary_txn_duration_s":0.0,"db_main_txn_duration_s":0.006,"db_ci_txn_duration_s":0.0,"db_main_replica_txn_duration_s":0.0,"db_ci_replica_txn_duration_s":0.0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.008,"db_main_duration_s":0.008,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"external_http_count":3,"external_http_duration_s":0.007039506017463282,"cpu_s":0.06638,"mem_objects":11710,"mem_bytes":762920,"mem_mallocs":2780,"mem_total_bytes":1231320,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.090109,"completed_at":"2024-05-03T05:10:52.164Z","load_balancing_strategy":"primary","db_duration_s":0.009304,"urgency":"low","target_duration_s":300,"target_scheduling_latency_s":60}
{"severity":"INFO","time":"2024-05-03T05:10:52.166Z","retry":0,"queue":"default","version":0,"store":null,"status_expiration":1800,"queue_namespace":"container_repository_delete","args":[],"class":"ContainerRegistry::DeleteContainerRepositoryWorker","jid":"50f5fed453b70bdc30159220","created_at":"2024-05-03T05:10:52.159Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"b3cc39cfff4d3c2553fb73cea1680888","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":"2024-05-03T05:10:52.160Z","job_size_bytes":2,"pid":503,"message":"ContainerRegistry::DeleteContainerRepositoryWorker JID-50f5fed453b70bdc30159220: start","job_status":"start","scheduling_latency_s":0.005232}
I can't seem to stop it.
I think the problem can be reproduced by these steps:
- Have Project 1 in Group A
- Have some containers in Project 1
- Schedule a container for deletion in Project 1
- Move Project 1 to Group B
- Wait for the clean up task to come around The clean up task fails to find the group, dies and repeats
Example Project
I don't know how to make this available.
What is the current bug behavior?
My CPU is maxed out because this task doesn't end correctly
What is the expected correct behavior?
This task should either realize it doesn't need to run, or run, fail and not run again
Relevant logs and/or screenshots
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2024-05-03T15:50:03.302Z","class":"ContainerRegistry::DeleteContainerRepositoryWorker","project_id":4,"container_repository_id":1,"container_repository_path":"home/clarkweb","tags_size_before_delete":1,"deleted_tags_size":null,"meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","job_status":"running","queue":"default","jid":"b4955609977b499cbc4baffe","retry":0}
{"severity":"INFO","time":"2024-05-03T15:50:03.313Z","retry":0,"queue":"default","version":0,"store":null,"status_expiration":1800,"queue_namespace":"container_repository_delete","args":[],"class":"ContainerRegistry::DeleteContainerRepositoryWorker","jid":"b4955609977b499cbc4baffe","created_at":"2024-05-03T15:50:03.219Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":"2024-05-03T15:50:03.221Z","job_size_bytes":2,"pid":506,"message":"ContainerRegistry::DeleteContainerRepositoryWorker JID-b4955609977b499cbc4baffe: done: 0.090851 sec","job_status":"done","scheduling_latency_s":0.001663,"redis_calls":9,"redis_duration_s":0.002288,"redis_read_bytes":10,"redis_write_bytes":1359,"redis_queues_calls":2,"redis_queues_duration_s":0.000279,"redis_queues_read_bytes":2,"redis_queues_write_bytes":615,"redis_queues_metadata_calls":2,"redis_queues_metadata_duration_s":0.000609,"redis_queues_metadata_read_bytes":3,"redis_queues_metadata_write_bytes":105,"redis_shared_state_calls":5,"redis_shared_state_duration_s":0.0014,"redis_shared_state_read_bytes":5,"redis_shared_state_write_bytes":639,"db_count":10,"db_write_count":3,"db_cached_count":1,"db_txn_count":1,"db_replica_txn_count":0,"db_primary_txn_count":0,"db_main_txn_count":1,"db_ci_txn_count":0,"db_main_replica_txn_count":0,"db_ci_replica_txn_count":0,"db_replica_count":0,"db_primary_count":10,"db_main_count":10,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_main_cached_count":1,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_txn_max_duration_s":0.0,"db_primary_txn_max_duration_s":0.0,"db_main_txn_max_duration_s":0.006,"db_ci_txn_max_duration_s":0.0,"db_main_replica_txn_max_duration_s":0.0,"db_ci_replica_txn_max_duration_s":0.0,"db_replica_txn_duration_s":0.0,"db_primary_txn_duration_s":0.0,"db_main_txn_duration_s":0.006,"db_ci_txn_duration_s":0.0,"db_main_replica_txn_duration_s":0.0,"db_ci_replica_txn_duration_s":0.0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.007,"db_main_duration_s":0.007,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"external_http_count":3,"external_http_duration_s":0.006910224998136982,"cpu_s":0.067325,"mem_objects":11710,"mem_bytes":977528,"mem_mallocs":4795,"mem_total_bytes":1445928,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.090851,"completed_at":"2024-05-03T15:50:03.313Z","load_balancing_strategy":"primary","db_duration_s":0.009586,"urgency":"low","target_duration_s":300,"target_scheduling_latency_s":60}
{"severity":"INFO","time":"2024-05-03T15:50:03.314Z","retry":0,"queue":"default","version":0,"store":null,"status_expiration":1800,"queue_namespace":"container_repository_delete","args":[],"class":"ContainerRegistry::DeleteContainerRepositoryWorker","jid":"d81d5f28c86e31f9ac11d6b1","created_at":"2024-05-03T15:50:03.309Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":"2024-05-03T15:50:03.310Z","job_size_bytes":2,"pid":506,"message":"ContainerRegistry::DeleteContainerRepositoryWorker JID-d81d5f28c86e31f9ac11d6b1: start","job_status":"start","scheduling_latency_s":0.003992}
==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"INFO","time":"2024-05-03T15:50:03.334Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","container_repository_id":1,"container_repository_path":"home/clarkweb","project_id":4,"third_party_cleanup_tags_service":true}
{"severity":"ERROR","time":"2024-05-03T15:50:03.392Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","service_class":"Projects::ContainerRepository::DeleteTagsService","container_repository_id":1,"project_id":4,"message":"could not delete tags: latest"}
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2024-05-03T15:50:03.392Z","class":"ContainerRegistry::DeleteContainerRepositoryWorker","project_id":4,"container_repository_id":1,"container_repository_path":"home/clarkweb","tags_size_before_delete":1,"deleted_tags_size":null,"meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","job_status":"running","queue":"default","jid":"d81d5f28c86e31f9ac11d6b1","retry":0}
{"severity":"INFO","time":"2024-05-03T15:50:03.402Z","retry":0,"queue":"default","version":0,"store":null,"status_expiration":1800,"queue_namespace":"container_repository_delete","args":[],"class":"ContainerRegistry::DeleteContainerRepositoryWorker","jid":"de577b3166e309d737b7a634","created_at":"2024-05-03T15:50:03.399Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":"2024-05-03T15:50:03.401Z","job_size_bytes":2,"pid":506,"message":"ContainerRegistry::DeleteContainerRepositoryWorker JID-de577b3166e309d737b7a634: start","job_status":"start","scheduling_latency_s":0.001148}
{"severity":"INFO","time":"2024-05-03T15:50:03.405Z","retry":0,"queue":"default","version":0,"store":null,"status_expiration":1800,"queue_namespace":"container_repository_delete","args":[],"class":"ContainerRegistry::DeleteContainerRepositoryWorker","jid":"d81d5f28c86e31f9ac11d6b1","created_at":"2024-05-03T15:50:03.309Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":"2024-05-03T15:50:03.310Z","job_size_bytes":2,"pid":506,"message":"ContainerRegistry::DeleteContainerRepositoryWorker JID-d81d5f28c86e31f9ac11d6b1: done: 0.090715 sec","job_status":"done","scheduling_latency_s":0.003992,"redis_calls":9,"redis_duration_s":0.002552,"redis_read_bytes":10,"redis_write_bytes":1358,"redis_queues_calls":2,"redis_queues_duration_s":0.000492,"redis_queues_read_bytes":2,"redis_queues_write_bytes":614,"redis_queues_metadata_calls":2,"redis_queues_metadata_duration_s":0.000955,"redis_queues_metadata_read_bytes":3,"redis_queues_metadata_write_bytes":105,"redis_shared_state_calls":5,"redis_shared_state_duration_s":0.001105,"redis_shared_state_read_bytes":5,"redis_shared_state_write_bytes":639,"db_count":10,"db_write_count":3,"db_cached_count":1,"db_txn_count":1,"db_replica_txn_count":0,"db_primary_txn_count":0,"db_main_txn_count":1,"db_ci_txn_count":0,"db_main_replica_txn_count":0,"db_ci_replica_txn_count":0,"db_replica_count":0,"db_primary_count":10,"db_main_count":10,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_main_cached_count":1,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_txn_max_duration_s":0.0,"db_primary_txn_max_duration_s":0.0,"db_main_txn_max_duration_s":0.006,"db_ci_txn_max_duration_s":0.0,"db_main_replica_txn_max_duration_s":0.0,"db_ci_replica_txn_max_duration_s":0.0,"db_replica_txn_duration_s":0.0,"db_primary_txn_duration_s":0.0,"db_main_txn_duration_s":0.006,"db_ci_txn_duration_s":0.0,"db_main_replica_txn_duration_s":0.0,"db_ci_replica_txn_duration_s":0.0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.008,"db_main_duration_s":0.008,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"external_http_count":3,"external_http_duration_s":0.0070094280235935,"cpu_s":0.069316,"mem_objects":11710,"mem_bytes":977528,"mem_mallocs":4795,"mem_total_bytes":1445928,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.090715,"completed_at":"2024-05-03T15:50:03.405Z","load_balancing_strategy":"primary","db_duration_s":0.008693,"urgency":"low","target_duration_s":300,"target_scheduling_latency_s":60}
==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"INFO","time":"2024-05-03T15:50:03.425Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","container_repository_id":1,"container_repository_path":"home/clarkweb","project_id":4,"third_party_cleanup_tags_service":true}
{"severity":"ERROR","time":"2024-05-03T15:50:03.486Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","service_class":"Projects::ContainerRepository::DeleteTagsService","container_repository_id":1,"project_id":4,"message":"could not delete tags: latest"}
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2024-05-03T15:50:03.486Z","class":"ContainerRegistry::DeleteContainerRepositoryWorker","project_id":4,"container_repository_id":1,"container_repository_path":"home/clarkweb","tags_size_before_delete":1,"deleted_tags_size":null,"meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","job_status":"running","queue":"default","jid":"de577b3166e309d737b7a634","retry":0}
{"severity":"INFO","time":"2024-05-03T15:50:03.495Z","retry":0,"queue":"default","version":0,"store":null,"status_expiration":1800,"queue_namespace":"container_repository_delete","args":[],"class":"ContainerRegistry::DeleteContainerRepositoryWorker","jid":"58eb5b74d032d1df90b094f2","created_at":"2024-05-03T15:50:03.493Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":"2024-05-03T15:50:03.494Z","job_size_bytes":2,"pid":506,"message":"ContainerRegistry::DeleteContainerRepositoryWorker JID-58eb5b74d032d1df90b094f2: start","job_status":"start","scheduling_latency_s":0.001125}
{"severity":"INFO","time":"2024-05-03T15:50:03.498Z","retry":0,"queue":"default","version":0,"store":null,"status_expiration":1800,"queue_namespace":"container_repository_delete","args":[],"class":"ContainerRegistry::DeleteContainerRepositoryWorker","jid":"de577b3166e309d737b7a634","created_at":"2024-05-03T15:50:03.399Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":"2024-05-03T15:50:03.401Z","job_size_bytes":2,"pid":506,"message":"ContainerRegistry::DeleteContainerRepositoryWorker JID-de577b3166e309d737b7a634: done: 0.095945 sec","job_status":"done","scheduling_latency_s":0.001148,"redis_calls":9,"redis_duration_s":0.002504,"redis_read_bytes":10,"redis_write_bytes":1359,"redis_queues_calls":2,"redis_queues_duration_s":0.000422,"redis_queues_read_bytes":2,"redis_queues_write_bytes":615,"redis_queues_metadata_calls":2,"redis_queues_metadata_duration_s":0.000927,"redis_queues_metadata_read_bytes":3,"redis_queues_metadata_write_bytes":105,"redis_shared_state_calls":5,"redis_shared_state_duration_s":0.001155,"redis_shared_state_read_bytes":5,"redis_shared_state_write_bytes":639,"db_count":10,"db_write_count":3,"db_cached_count":1,"db_txn_count":1,"db_replica_txn_count":0,"db_primary_txn_count":0,"db_main_txn_count":1,"db_ci_txn_count":0,"db_main_replica_txn_count":0,"db_ci_replica_txn_count":0,"db_replica_count":0,"db_primary_count":10,"db_main_count":10,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_main_cached_count":1,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_txn_max_duration_s":0.0,"db_primary_txn_max_duration_s":0.0,"db_main_txn_max_duration_s":0.005,"db_ci_txn_max_duration_s":0.0,"db_main_replica_txn_max_duration_s":0.0,"db_ci_replica_txn_max_duration_s":0.0,"db_replica_txn_duration_s":0.0,"db_primary_txn_duration_s":0.0,"db_main_txn_duration_s":0.005,"db_ci_txn_duration_s":0.0,"db_main_replica_txn_duration_s":0.0,"db_ci_replica_txn_duration_s":0.0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.007,"db_main_duration_s":0.007,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"external_http_count":3,"external_http_duration_s":0.006635965983150527,"cpu_s":0.067656,"mem_objects":11710,"mem_bytes":977528,"mem_mallocs":4795,"mem_total_bytes":1445928,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.095945,"completed_at":"2024-05-03T15:50:03.498Z","load_balancing_strategy":"primary","db_duration_s":0.008502,"urgency":"low","target_duration_s":300,"target_scheduling_latency_s":60}
==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"INFO","time":"2024-05-03T15:50:03.517Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","container_repository_id":1,"container_repository_path":"home/clarkweb","project_id":4,"third_party_cleanup_tags_service":true}
{"severity":"ERROR","time":"2024-05-03T15:50:03.581Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"8f1e969d41a69f951fd74fd5be26b157","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","service_class":"Projects::ContainerRepository::DeleteTagsService","container_repository_id":1,"project_id":4,"message":"could not delete tags: latest"}
Output of checks
I'm not on gitlab.com. It's a self hosted container of 16.11.0-ce
Results of GitLab environment info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 3.1.4p223
Gem Version: 3.5.7
Bundler Version:2.5.8
Rake Version: 13.0.6
Redis Version: 7.0.15
Sidekiq Version:7.1.6
Go Version: unknown
GitLab information
Version: 16.11.0
Revision: 0454e1accb3
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 14.11
URL: https://git.azkablan.com:30000
HTTP Clone URL: https://git.azkablan.com:30000/some-group/some-project.git
SSH Clone URL: ssh://git@git.azkablan.com:30022/some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.35.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Gitaly
- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 16.11.0
- default Git Version: 2.43.2
Results of GitLab application Check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.35.0 ? ... OK (14.35.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
110/2 ... yes
4/4 ... yes
4/5 ... yes
6/6 ... yes
5/7 ... yes
5/8 ... yes
6/9 ... yes
8/10 ... yes
110/11 ... yes
7/12 ... yes
8/13 ... yes
9/14 ... yes
7/15 ... yes
7/16 ... yes
8/17 ... yes
9/18 ... yes
8/19 ... yes
4/20 ... yes
8/21 ... yes
7/22 ... yes
4/23 ... yes
8/24 ... yes
4/25 ... yes
4/26 ... yes
7/27 ... yes
4/28 ... yes
7/29 ... yes
4/30 ... yes
7/31 ... yes
7/32 ... yes
4/33 ... yes
4/34 ... yes
4/35 ... yes
4/36 ... yes
7/37 ... yes
7/38 ... yes
7/39 ... yes
4/40 ... yes
7/41 ... yes
4/42 ... yes
7/43 ... yes
4/44 ... yes
7/45 ... yes
4/46 ... yes
7/47 ... yes
7/48 ... yes
4/49 ... yes
7/50 ... yes
4/51 ... yes
4/52 ... yes
7/53 ... yes
4/54 ... yes
7/55 ... yes
7/56 ... yes
4/57 ... yes
4/58 ... yes
4/59 ... yes
4/60 ... yes
4/61 ... yes
4/62 ... yes
4/63 ... yes
4/64 ... yes
4/65 ... yes
4/66 ... yes
4/67 ... yes
4/68 ... yes
4/69 ... yes
4/70 ... yes
4/71 ... yes
4/72 ... yes
4/73 ... yes
4/74 ... yes
4/75 ... yes
4/76 ... yes
4/77 ... yes
4/78 ... yes
4/79 ... yes
4/80 ... yes
4/81 ... yes
4/82 ... yes
4/83 ... yes
4/84 ... yes
4/85 ... yes
4/86 ... yes
4/87 ... yes
4/88 ... yes
4/89 ... yes
4/90 ... yes
4/91 ... yes
4/92 ... yes
4/93 ... yes
4/94 ... yes
4/95 ... yes
4/96 ... yes
4/97 ... yes
106/98 ... yes
106/99 ... yes
106/100 ... yes
4/104 ... yes
110/105 ... yes
110/106 ... yes
110/107 ... yes
6/108 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.1.4)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Nothing I've tried fixes it. I've tried disabling registries, which works for a few minutes but then it tries running the task again.