installer fails on Debian 11 / broken alertmanager
Since this morning at 11:11:16 CET, our GitLab server misbehaves and responds 500 Internal Server Error to all requests.
GitLab version: 15.6.2 (EE) on Debian 11
Restart and reboot had no effect.
It is unclear to me why GitLab stopped working at that exact point in time...
error logs
==> /var/log/gitlab/alertmanager/current <==
2022-12-20_19:03:09.18316 ts=2022-12-20T19:03:09.183Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=master, revision=)"
2022-12-20_19:03:09.18323 ts=2022-12-20T19:03:09.183Z caller=main.go:232 level=info build_context="(go=go1.18.7, user=GitLab-Omnibus, date=)"
2022-12-20_19:03:09.18489 ts=2022-12-20T19:03:09.184Z caller=cluster.go:178 level=warn component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided"
2022-12-20_19:03:09.18680 ts=2022-12-20T19:03:09.186Z caller=main.go:263 level=error msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"other logs right before the event
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.709Z","retry":0,"queue":"cronjob:incident_management_pending_escalations_schedule_check_cron","version":0,"queue_namespace":"cronjob","args":[],"class":"IncidentManagement::PendingEscalations::ScheduleCheckCronWorker","jid":"41b12acf15d02052e377d7ae","created_at":"2022-12-20T11:11:04.704Z","meta.caller_id":"Cronjob","correlation_id":"83ea9f12b81d4e2e17071d14a62b70a2","meta.root_caller_id":"Cronjob","meta.feature_category":"incident_management","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:incident_management_pending_escalations_schedule_check_cron:979447f3f3db1dd62a2679e7ff5835fcbb6ccf42394193a5feb300b5ecfe13e6","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.708Z","job_size_bytes":2,"pid":3618946,"message":"IncidentManagement::PendingEscalations::ScheduleCheckCronWorker JID-41b12acf15d02052e377d7ae: start","job_status":"start","scheduling_latency_s":0.001144}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.748Z","retry":0,"queue":"cronjob:incident_management_pending_escalations_schedule_check_cron","version":0,"queue_namespace":"cronjob","args":[],"class":"IncidentManagement::PendingEscalations::ScheduleCheckCronWorker","jid":"41b12acf15d02052e377d7ae","created_at":"2022-12-20T11:11:04.704Z","meta.caller_id":"Cronjob","correlation_id":"83ea9f12b81d4e2e17071d14a62b70a2","meta.root_caller_id":"Cronjob","meta.feature_category":"incident_management","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:incident_management_pending_escalations_schedule_check_cron:979447f3f3db1dd62a2679e7ff5835fcbb6ccf42394193a5feb300b5ecfe13e6","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.708Z","job_size_bytes":2,"pid":3618946,"message":"IncidentManagement::PendingEscalations::ScheduleCheckCronWorker JID-41b12acf15d02052e377d7ae: done: 0.038829 sec","job_status":"done","scheduling_latency_s":0.001144,"redis_calls":2,"redis_duration_s":0.030967,"redis_read_bytes":2,"redis_write_bytes":246,"redis_queues_calls":2,"redis_queues_duration_s":0.030967,"redis_queues_read_bytes":2,"redis_queues_write_bytes":246,"db_count":2,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":2,"db_main_count":2,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.001,"db_main_duration_s":0.001,"db_main_replica_duration_s":0.0,"cpu_s":0.004961,"mem_objects":2029,"mem_bytes":126488,"mem_mallocs":460,"mem_total_bytes":207648,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.038829,"completed_at":"2022-12-20T11:11:04.748Z","load_balancing_strategy":"primary","db_duration_s":0.001224}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.750Z","retry":0,"queue":"cronjob:user_status_cleanup_batch","version":0,"queue_namespace":"cronjob","args":[],"class":"UserStatusCleanup::BatchWorker","jid":"adae27287a844c227978b144","created_at":"2022-12-20T11:11:04.742Z","meta.caller_id":"Cronjob","correlation_id":"0c21fe625d5e352c43611e0c0102de3a","meta.root_caller_id":"Cronjob","meta.feature_category":"users","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:user_status_cleanup_batch:63596f84f2aa01599e5d284b58c5b497d0d152817e7aa09bae73984558ee32bf","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.744Z","job_size_bytes":2,"pid":3618946,"message":"UserStatusCleanup::BatchWorker JID-adae27287a844c227978b144: start","job_status":"start","scheduling_latency_s":0.00581}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.759Z","retry":0,"queue":"cronjob:loose_foreign_keys_cleanup","version":0,"queue_namespace":"cronjob","args":[],"class":"LooseForeignKeys::CleanupWorker","jid":"d586831f8cf6ecdc9db932a7","created_at":"2022-12-20T11:11:04.756Z","meta.caller_id":"Cronjob","correlation_id":"c3978e7265bb7131aea7b4aa1221832c","meta.root_caller_id":"Cronjob","meta.feature_category":"pods","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:loose_foreign_keys_cleanup:2180384c8554b410820afa81ad3479301139c7627524f882c6bca44488742df9","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.757Z","job_size_bytes":2,"pid":3618946,"message":"LooseForeignKeys::CleanupWorker JID-d586831f8cf6ecdc9db932a7: start","job_status":"start","scheduling_latency_s":0.002055}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.778Z","retry":0,"queue":"cronjob:user_status_cleanup_batch","version":0,"queue_namespace":"cronjob","args":[],"class":"UserStatusCleanup::BatchWorker","jid":"adae27287a844c227978b144","created_at":"2022-12-20T11:11:04.742Z","meta.caller_id":"Cronjob","correlation_id":"0c21fe625d5e352c43611e0c0102de3a","meta.root_caller_id":"Cronjob","meta.feature_category":"users","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:user_status_cleanup_batch:63596f84f2aa01599e5d284b58c5b497d0d152817e7aa09bae73984558ee32bf","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.744Z","job_size_bytes":2,"pid":3618946,"message":"UserStatusCleanup::BatchWorker JID-adae27287a844c227978b144: done: 0.028809 sec","job_status":"done","scheduling_latency_s":0.00581,"redis_calls":2,"redis_duration_s":0.002873,"redis_read_bytes":2,"redis_write_bytes":212,"redis_queues_calls":2,"redis_queues_duration_s":0.002873,"redis_queues_read_bytes":2,"redis_queues_write_bytes":212,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.015,"db_main_duration_s":0.015,"db_main_replica_duration_s":0.0,"cpu_s":0.003707,"mem_objects":1525,"mem_bytes":113968,"mem_mallocs":332,"mem_total_bytes":174968,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.028809,"completed_at":"2022-12-20T11:11:04.778Z","load_balancing_strategy":"primary","db_duration_s":0.014758}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.782Z","retry":0,"queue":"cronjob:database_batched_background_migration_ci_database","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigration::CiDatabaseWorker","jid":"66ce35fa3aed62d9216765eb","created_at":"2022-12-20T11:11:04.776Z","meta.caller_id":"Cronjob","correlation_id":"dcb1fc1df37885c2533c98ca024f642f","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:database_batched_background_migration_ci_database:6ba8adee4a8c1e77d2f087a2765c43226ceffa1fd65abc34b95725a7c9abd857","enqueued_at":"2022-12-20T11:11:04.779Z","job_size_bytes":2,"pid":3618946,"message":"Database::BatchedBackgroundMigration::CiDatabaseWorker JID-66ce35fa3aed62d9216765eb: start","job_status":"start","scheduling_latency_s":0.002972}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.786Z","class":"Database::BatchedBackgroundMigration::CiDatabaseWorker","database":"ci","message":"skipping migration execution for unconfigured database","retry":0}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.795Z","retry":0,"queue":"cronjob:database_batched_background_migration_ci_database","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigration::CiDatabaseWorker","jid":"66ce35fa3aed62d9216765eb","created_at":"2022-12-20T11:11:04.776Z","meta.caller_id":"Cronjob","correlation_id":"dcb1fc1df37885c2533c98ca024f642f","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:database_batched_background_migration_ci_database:6ba8adee4a8c1e77d2f087a2765c43226ceffa1fd65abc34b95725a7c9abd857","enqueued_at":"2022-12-20T11:11:04.779Z","job_size_bytes":2,"pid":3618946,"message":"Database::BatchedBackgroundMigration::CiDatabaseWorker JID-66ce35fa3aed62d9216765eb: done: 0.013433 sec","job_status":"done","scheduling_latency_s":0.002972,"redis_calls":2,"redis_duration_s":0.008207,"redis_read_bytes":2,"redis_write_bytes":236,"redis_queues_calls":2,"redis_queues_duration_s":0.008207,"redis_queues_read_bytes":2,"redis_queues_write_bytes":236,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.002151,"mem_objects":1282,"mem_bytes":108552,"mem_mallocs":285,"mem_total_bytes":159832,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.013433,"completed_at":"2022-12-20T11:11:04.795Z","load_balancing_strategy":"primary","db_duration_s":0.0}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.801Z","retry":0,"queue":"cronjob:schedule_merge_request_cleanup_refs","version":0,"queue_namespace":"cronjob","args":[],"class":"ScheduleMergeRequestCleanupRefsWorker","jid":"00503f121aba3b291a5576a5","created_at":"2022-12-20T11:11:04.796Z","meta.caller_id":"Cronjob","correlation_id":"c2a090a531715644dae6b14c0dd9eb11","meta.root_caller_id":"Cronjob","meta.feature_category":"code_review","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:schedule_merge_request_cleanup_refs:33e8a9dcd4c9780ad0ea123ad7ccbabde1aa1e90ffcbb928434ba4b5800a5811","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.797Z","job_size_bytes":2,"pid":3618946,"message":"ScheduleMergeRequestCleanupRefsWorker JID-00503f121aba3b291a5576a5: start","job_status":"start","scheduling_latency_s":0.003825}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.857Z","retry":0,"queue":"cronjob:schedule_merge_request_cleanup_refs","version":0,"queue_namespace":"cronjob","args":[],"class":"ScheduleMergeRequestCleanupRefsWorker","jid":"00503f121aba3b291a5576a5","created_at":"2022-12-20T11:11:04.796Z","meta.caller_id":"Cronjob","correlation_id":"c2a090a531715644dae6b14c0dd9eb11","meta.root_caller_id":"Cronjob","meta.feature_category":"code_review","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:schedule_merge_request_cleanup_refs:33e8a9dcd4c9780ad0ea123ad7ccbabde1aa1e90ffcbb928434ba4b5800a5811","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.797Z","job_size_bytes":2,"pid":3618946,"message":"ScheduleMergeRequestCleanupRefsWorker JID-00503f121aba3b291a5576a5: done: 0.056347 sec","job_status":"done","scheduling_latency_s":0.003825,"redis_calls":2,"redis_duration_s":0.049593,"redis_read_bytes":2,"redis_write_bytes":222,"redis_queues_calls":2,"redis_queues_duration_s":0.049593,"redis_queues_read_bytes":2,"redis_queues_write_bytes":222,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.004402,"mem_objects":2687,"mem_bytes":336152,"mem_mallocs":950,"mem_total_bytes":443632,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.056347,"completed_at":"2022-12-20T11:11:04.857Z","load_balancing_strategy":"primary","db_duration_s":0.0}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.862Z","retry":0,"queue":"cronjob:geo_sidekiq_cron_config","version":0,"queue_namespace":"cronjob","args":[],"class":"Geo::SidekiqCronConfigWorker","jid":"4565a6d3446470c7139ff534","created_at":"2022-12-20T11:11:04.858Z","meta.caller_id":"Cronjob","correlation_id":"6b86325b63c1b066d457a998489ef95b","meta.root_caller_id":"Cronjob","meta.feature_category":"geo_replication","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:geo_sidekiq_cron_config:ee3c9c9863d06b67487cd0dcfc522a28b6dc54e34bf5abd078467931033818a0","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.860Z","job_size_bytes":2,"pid":3618946,"message":"Geo::SidekiqCronConfigWorker JID-4565a6d3446470c7139ff534: start","job_status":"start","scheduling_latency_s":0.002203}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.873Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexInitialBulkCronWorker","jid":"c295796db2f936334a90a185","created_at":"2022-12-20T11:11:04.870Z","meta.caller_id":"Cronjob","correlation_id":"fa9835b9790356ce88dbae249fc13a6c","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:ba175455ba4249d8c954504ae77a1c3684a61c72005cc3a3032e14c625ef4d8a","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.871Z","job_size_bytes":2,"pid":3618946,"message":"ElasticIndexInitialBulkCronWorker JID-c295796db2f936334a90a185: start","job_status":"start","scheduling_latency_s":0.002039}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.901Z","retry":0,"queue":"cronjob:update_all_mirrors","version":0,"queue_namespace":"cronjob","args":[],"class":"UpdateAllMirrorsWorker","jid":"115c0e54ae4631021755f3e7","created_at":"2022-12-20T11:11:04.895Z","meta.caller_id":"Cronjob","correlation_id":"02137c2439d1203f110cfeb7e724895f","meta.root_caller_id":"Cronjob","meta.feature_category":"source_code_management","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:update_all_mirrors:06f0ae3fb264ecea0081f8b5bcdfdf63d385e3db07629cf8d60f5d6f216854cc","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.899Z","job_size_bytes":2,"pid":3618946,"message":"UpdateAllMirrorsWorker JID-115c0e54ae4631021755f3e7: start","job_status":"start","scheduling_latency_s":0.001876}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.909Z","retry":0,"queue":"cronjob:loose_foreign_keys_cleanup","version":0,"queue_namespace":"cronjob","args":[],"class":"LooseForeignKeys::CleanupWorker","jid":"d586831f8cf6ecdc9db932a7","created_at":"2022-12-20T11:11:04.756Z","meta.caller_id":"Cronjob","correlation_id":"c3978e7265bb7131aea7b4aa1221832c","meta.root_caller_id":"Cronjob","meta.feature_category":"pods","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:loose_foreign_keys_cleanup:2180384c8554b410820afa81ad3479301139c7627524f882c6bca44488742df9","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.757Z","job_size_bytes":2,"pid":3618946,"message":"LooseForeignKeys::CleanupWorker JID-d586831f8cf6ecdc9db932a7: done: 0.150094 sec","job_status":"done","scheduling_latency_s":0.002055,"redis_calls":4,"redis_duration_s":0.009776,"redis_read_bytes":4,"redis_write_bytes":514,"redis_queues_calls":2,"redis_queues_duration_s":0.007337,"redis_queues_read_bytes":2,"redis_queues_write_bytes":213,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.002439,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":301,"db_count":9,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":9,"db_main_count":9,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.027,"db_main_duration_s":0.027,"db_main_replica_duration_s":0.0,"cpu_s":0.013644,"mem_objects":5694,"mem_bytes":311784,"mem_mallocs":1131,"mem_total_bytes":539544,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.loose_foreign_keys_cleanup_worker.stats":{"over_limit":false,"delete_count_by_table":{},"update_count_by_table":{},"delete_count":0,"update_count":0,"connection":"main"},"duration_s":0.150094,"completed_at":"2022-12-20T11:11:04.909Z","load_balancing_strategy":"primary","db_duration_s":0.111702}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.952Z","retry":0,"queue":"cronjob:database_batched_background_migration","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigrationWorker","jid":"fa84d5f625dfcf0cfd678321","created_at":"2022-12-20T11:11:04.946Z","meta.caller_id":"Cronjob","correlation_id":"c91a2f1a968b45b8aff0881878bd0404","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:database_batched_background_migration:592d9619e1997b640b70ce6a22f6713bc7793bb7a4e342b7380d90b691fcd6ae","enqueued_at":"2022-12-20T11:11:04.948Z","job_size_bytes":2,"pid":3618946,"message":"Database::BatchedBackgroundMigrationWorker JID-fa84d5f625dfcf0cfd678321: start","job_status":"start","scheduling_latency_s":0.004295}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.953Z","retry":0,"queue":"cronjob:update_all_mirrors","version":0,"queue_namespace":"cronjob","args":[],"class":"UpdateAllMirrorsWorker","jid":"115c0e54ae4631021755f3e7","created_at":"2022-12-20T11:11:04.895Z","meta.caller_id":"Cronjob","correlation_id":"02137c2439d1203f110cfeb7e724895f","meta.root_caller_id":"Cronjob","meta.feature_category":"source_code_management","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:update_all_mirrors:06f0ae3fb264ecea0081f8b5bcdfdf63d385e3db07629cf8d60f5d6f216854cc","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.899Z","job_size_bytes":2,"pid":3618946,"message":"UpdateAllMirrorsWorker JID-115c0e54ae4631021755f3e7: done: 0.052344 sec","job_status":"done","scheduling_latency_s":0.001876,"redis_calls":6,"redis_duration_s":0.037378,"redis_read_bytes":6,"redis_write_bytes":531,"redis_queues_calls":2,"redis_queues_duration_s":0.003053,"redis_queues_read_bytes":2,"redis_queues_write_bytes":205,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.034325,"redis_shared_state_read_bytes":4,"redis_shared_state_write_bytes":326,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"db_main_duration_s":0.005,"db_main_replica_duration_s":0.0,"cpu_s":0.00787,"mem_objects":3790,"mem_bytes":502088,"mem_mallocs":1186,"mem_total_bytes":653688,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.052344,"completed_at":"2022-12-20T11:11:04.953Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.005114}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.956Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexInitialBulkCronWorker","jid":"c295796db2f936334a90a185","created_at":"2022-12-20T11:11:04.870Z","meta.caller_id":"Cronjob","correlation_id":"fa9835b9790356ce88dbae249fc13a6c","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:ba175455ba4249d8c954504ae77a1c3684a61c72005cc3a3032e14c625ef4d8a","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.871Z","job_size_bytes":2,"pid":3618946,"message":"ElasticIndexInitialBulkCronWorker JID-c295796db2f936334a90a185: done: 0.082866 sec","job_status":"done","scheduling_latency_s":0.002039,"redis_calls":20,"redis_duration_s":0.058227,"redis_read_bytes":4,"redis_write_bytes":1624,"redis_queues_calls":2,"redis_queues_duration_s":0.000859,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":18,"redis_shared_state_duration_s":0.057368,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":1406,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.017,"db_main_duration_s":0.017,"db_main_replica_duration_s":0.0,"cpu_s":0.006169,"mem_objects":3188,"mem_bytes":847056,"mem_mallocs":574,"mem_total_bytes":974576,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"duration_s":0.082866,"completed_at":"2022-12-20T11:11:04.956Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.016995}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.981Z","retry":0,"queue":"cronjob:elastic_index_bulk_cron","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexBulkCronWorker","jid":"49452f476c361b6a3a819aa6","created_at":"2022-12-20T11:11:04.976Z","meta.caller_id":"Cronjob","correlation_id":"e8c094c8c09318f5267edccbafb6bcc6","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_bulk_cron:f252f68f3cc1cae1877f9a0e1f5b889102a68a9e335a3a5a9e683c7bdf0507f5","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.979Z","job_size_bytes":2,"pid":3618946,"message":"ElasticIndexBulkCronWorker JID-49452f476c361b6a3a819aa6: start","job_status":"start","scheduling_latency_s":0.002591}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:04.985Z","retry":0,"queue":"cronjob:database_batched_background_migration","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigrationWorker","jid":"fa84d5f625dfcf0cfd678321","created_at":"2022-12-20T11:11:04.946Z","meta.caller_id":"Cronjob","correlation_id":"c91a2f1a968b45b8aff0881878bd0404","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:database_batched_background_migration:592d9619e1997b640b70ce6a22f6713bc7793bb7a4e342b7380d90b691fcd6ae","enqueued_at":"2022-12-20T11:11:04.948Z","job_size_bytes":2,"pid":3618946,"message":"Database::BatchedBackgroundMigrationWorker JID-fa84d5f625dfcf0cfd678321: done: 0.033054 sec","job_status":"done","scheduling_latency_s":0.004295,"redis_calls":2,"redis_duration_s":0.00366,"redis_read_bytes":2,"redis_write_bytes":224,"redis_queues_calls":2,"redis_queues_duration_s":0.00366,"redis_queues_read_bytes":2,"redis_queues_write_bytes":224,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"db_main_duration_s":0.005,"db_main_replica_duration_s":0.0,"cpu_s":0.003908,"mem_objects":1950,"mem_bytes":122432,"mem_mallocs":417,"mem_total_bytes":200432,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.033054,"completed_at":"2022-12-20T11:11:04.985Z","load_balancing_strategy":"primary","db_duration_s":0.005039}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:05.145Z","retry":0,"queue":"cronjob:geo_sidekiq_cron_config","version":0,"queue_namespace":"cronjob","args":[],"class":"Geo::SidekiqCronConfigWorker","jid":"4565a6d3446470c7139ff534","created_at":"2022-12-20T11:11:04.858Z","meta.caller_id":"Cronjob","correlation_id":"6b86325b63c1b066d457a998489ef95b","meta.root_caller_id":"Cronjob","meta.feature_category":"geo_replication","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:geo_sidekiq_cron_config:ee3c9c9863d06b67487cd0dcfc522a28b6dc54e34bf5abd078467931033818a0","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.860Z","job_size_bytes":2,"pid":3618946,"message":"Geo::SidekiqCronConfigWorker JID-4565a6d3446470c7139ff534: done: 0.283364 sec","job_status":"done","scheduling_latency_s":0.002203,"redis_calls":152,"redis_duration_s":0.258935,"redis_read_bytes":59126,"redis_write_bytes":10162,"redis_queues_calls":152,"redis_queues_duration_s":0.258935,"redis_queues_read_bytes":59126,"redis_queues_write_bytes":10162,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.159217,"mem_objects":94064,"mem_bytes":82583696,"mem_mallocs":25393,"mem_total_bytes":86346256,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.283364,"completed_at":"2022-12-20T11:11:05.145Z","load_balancing_strategy":"primary","db_duration_s":0.0}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:05.152Z","retry":0,"queue":"cronjob:elastic_index_bulk_cron","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexBulkCronWorker","jid":"49452f476c361b6a3a819aa6","created_at":"2022-12-20T11:11:04.976Z","meta.caller_id":"Cronjob","correlation_id":"e8c094c8c09318f5267edccbafb6bcc6","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_bulk_cron:f252f68f3cc1cae1877f9a0e1f5b889102a68a9e335a3a5a9e683c7bdf0507f5","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.979Z","job_size_bytes":2,"pid":3618946,"message":"ElasticIndexBulkCronWorker JID-49452f476c361b6a3a819aa6: done: 0.170527 sec","job_status":"done","scheduling_latency_s":0.002591,"redis_calls":20,"redis_duration_s":0.15266300000000002,"redis_read_bytes":4,"redis_write_bytes":1712,"redis_queues_calls":2,"redis_queues_duration_s":0.002121,"redis_queues_read_bytes":2,"redis_queues_write_bytes":210,"redis_shared_state_calls":18,"redis_shared_state_duration_s":0.150542,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":1502,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.009,"db_main_duration_s":0.009,"db_main_replica_duration_s":0.0,"cpu_s":0.006104,"mem_objects":3187,"mem_bytes":847616,"mem_mallocs":572,"mem_total_bytes":975096,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_bulk_cron_worker.records_count":0,"duration_s":0.170527,"completed_at":"2022-12-20T11:11:05.152Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.009407}
./gitlab/gitaly/current:{"command.count":1,"command.cpu_time_ms":1,"command.inblock":0,"command.majflt":0,"command.maxrss":327344,"command.minflt":141,"command.oublock":0,"command.real_time_ms":1,"command.system_time_ms":0,"command.user_time_ms":1,"correlation_id":"01GMQG6YSSF6VDXABEGKXWNPZT","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"FindCommit","grpc.request.deadline":"2022-12-20T11:11:05.100","grpc.request.fullMethod":"/gitaly.CommitService/FindCommit","grpc.request.glProjectPath":"ees/warehouse/server","grpc.request.glRepository":"project-9","grpc.request.payload_bytes":170,"grpc.request.repoPath":"@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":340,"grpc.service":"gitaly.CommitService","grpc.start_time":"2022-12-20T11:10:35.500","grpc.time_ms":2.116,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":3618734,"remote_ip":"130.75.237.111","span.kind":"server","system":"grpc","time":"2022-12-20T10:10:35.502Z","user_id":"47","username":"group_25_bot"}
./gitlab/gitaly/current:{"command.count":1,"command.cpu_time_ms":1,"command.inblock":0,"command.majflt":0,"command.maxrss":327344,"command.minflt":145,"command.oublock":0,"command.real_time_ms":1,"command.system_time_ms":0,"command.user_time_ms":1,"correlation_id":"01GMQG6YSEE5AA7ZJKH2T5XK1G","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"server_stream","grpc.method":"ListCommitsByOid","grpc.request.deadline":"2022-12-20T11:11:05.031","grpc.request.fullMethod":"/gitaly.CommitService/ListCommitsByOid","grpc.request.glProjectPath":"ees/warehouse/server","grpc.request.glRepository":"project-9","grpc.request.payload_bytes":170,"grpc.request.repoPath":"@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":340,"grpc.service":"gitaly.CommitService","grpc.start_time":"2022-12-20T11:10:35.532","grpc.time_ms":2.365,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":3618734,"remote_ip":"130.75.237.111","span.kind":"server","system":"grpc","time":"2022-12-20T10:10:35.534Z","user_id":"47","username":"group_25_bot"}
./gitlab/gitaly/current:{"command.count":1,"command.cpu_time_ms":1,"command.inblock":0,"command.majflt":0,"command.maxrss":327344,"command.minflt":143,"command.oublock":0,"command.real_time_ms":1,"command.system_time_ms":0,"command.user_time_ms":1,"correlation_id":"01GMQG7W4GA7TNKHDY3VGG2SJN","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"FindCommit","grpc.request.deadline":"2022-12-20T11:11:35.023","grpc.request.fullMethod":"/gitaly.CommitService/FindCommit","grpc.request.glProjectPath":"ees/warehouse/server","grpc.request.glRepository":"project-9","grpc.request.payload_bytes":170,"grpc.request.repoPath":"@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":340,"grpc.service":"gitaly.CommitService","grpc.start_time":"2022-12-20T11:11:05.523","grpc.time_ms":2.536,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":3618734,"remote_ip":"130.75.237.111","span.kind":"server","system":"grpc","time":"2022-12-20T10:11:05.526Z","user_id":"47","username":"group_25_bot"}
./gitlab/gitaly/current:{"command.count":1,"command.cpu_time_ms":1,"command.inblock":0,"command.majflt":0,"command.maxrss":327344,"command.minflt":142,"command.oublock":0,"command.real_time_ms":2,"command.system_time_ms":0,"command.user_time_ms":1,"correlation_id":"01GMQG7W4DZH25PVM0G40R8N3P","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"server_stream","grpc.method":"ListCommitsByOid","grpc.request.deadline":"2022-12-20T11:11:35.051","grpc.request.fullMethod":"/gitaly.CommitService/ListCommitsByOid","grpc.request.glProjectPath":"ees/warehouse/server","grpc.request.glRepository":"project-9","grpc.request.payload_bytes":170,"grpc.request.repoPath":"@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":340,"grpc.service":"gitaly.CommitService","grpc.start_time":"2022-12-20T11:11:05.551","grpc.time_ms":3.624,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":3618734,"remote_ip":"130.75.237.111","span.kind":"server","system":"grpc","time":"2022-12-20T10:11:05.555Z","user_id":"47","username":"group_25_bot"}
./gitlab/gitaly/current:{"command.count":1,"command.cpu_time_ms":1,"command.inblock":0,"command.majflt":0,"command.maxrss":327344,"command.minflt":143,"command.oublock":0,"command.real_time_ms":1,"command.system_time_ms":0,"command.user_time_ms":1,"correlation_id":"01GMQKNQYY4CP5N10X2ZR9SFE5","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"FindCommit","grpc.request.deadline":"2022-12-20T12:11:35.044","grpc.request.fullMethod":"/gitaly.CommitService/FindCommit","grpc.request.glProjectPath":"ees/warehouse/server","grpc.request.glRepository":"project-9","grpc.request.payload_bytes":170,"grpc.request.repoPath":"@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":340,"grpc.service":"gitaly.CommitService","grpc.start_time":"2022-12-20T12:11:05.744","grpc.time_ms":2.442,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":3618734,"remote_ip":"130.75.237.111","span.kind":"server","system":"grpc","time":"2022-12-20T11:11:05.747Z","user_id":"47","username":"group_25_bot"}
./gitlab/gitaly/current:{"command.count":1,"command.cpu_time_ms":1,"command.inblock":0,"command.majflt":0,"command.maxrss":327344,"command.minflt":143,"command.oublock":0,"command.real_time_ms":1,"command.system_time_ms":0,"command.user_time_ms":1,"correlation_id":"01GMQKNQYTKX19FANASYHRCSC3","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"server_stream","grpc.method":"ListCommitsByOid","grpc.request.deadline":"2022-12-20T12:11:35.056","grpc.request.fullMethod":"/gitaly.CommitService/ListCommitsByOid","grpc.request.glProjectPath":"ees/warehouse/server","grpc.request.glRepository":"project-9","grpc.request.payload_bytes":170,"grpc.request.repoPath":"@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":340,"grpc.service":"gitaly.CommitService","grpc.start_time":"2022-12-20T12:11:05.756","grpc.time_ms":2.233,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":3618734,"remote_ip":"130.75.237.111","span.kind":"server","system":"grpc","time":"2022-12-20T11:11:05.758Z","user_id":"47","username":"group_25_bot"}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:05.145Z","retry":0,"queue":"cronjob:geo_sidekiq_cron_config","version":0,"queue_namespace":"cronjob","args":[],"class":"Geo::SidekiqCronConfigWorker","jid":"4565a6d3446470c7139ff534","created_at":"2022-12-20T11:11:04.858Z","meta.caller_id":"Cronjob","correlation_id":"6b86325b63c1b066d457a998489ef95b","meta.root_caller_id":"Cronjob","meta.feature_category":"geo_replication","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:geo_sidekiq_cron_config:ee3c9c9863d06b67487cd0dcfc522a28b6dc54e34bf5abd078467931033818a0","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.860Z","job_size_bytes":2,"pid":3618946,"message":"Geo::SidekiqCronConfigWorker JID-4565a6d3446470c7139ff534: done: 0.283364 sec","job_status":"done","scheduling_latency_s":0.002203,"redis_calls":152,"redis_duration_s":0.258935,"redis_read_bytes":59126,"redis_write_bytes":10162,"redis_queues_calls":152,"redis_queues_duration_s":0.258935,"redis_queues_read_bytes":59126,"redis_queues_write_bytes":10162,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.159217,"mem_objects":94064,"mem_bytes":82583696,"mem_mallocs":25393,"mem_total_bytes":86346256,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.283364,"completed_at":"2022-12-20T11:11:05.145Z","load_balancing_strategy":"primary","db_duration_s":0.0}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:05.152Z","retry":0,"queue":"cronjob:elastic_index_bulk_cron","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexBulkCronWorker","jid":"49452f476c361b6a3a819aa6","created_at":"2022-12-20T11:11:04.976Z","meta.caller_id":"Cronjob","correlation_id":"e8c094c8c09318f5267edccbafb6bcc6","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_bulk_cron:f252f68f3cc1cae1877f9a0e1f5b889102a68a9e335a3a5a9e683c7bdf0507f5","size_limiter":"validated","enqueued_at":"2022-12-20T11:11:04.979Z","job_size_bytes":2,"pid":3618946,"message":"ElasticIndexBulkCronWorker JID-49452f476c361b6a3a819aa6: done: 0.170527 sec","job_status":"done","scheduling_latency_s":0.002591,"redis_calls":20,"redis_duration_s":0.15266300000000002,"redis_read_bytes":4,"redis_write_bytes":1712,"redis_queues_calls":2,"redis_queues_duration_s":0.002121,"redis_queues_read_bytes":2,"redis_queues_write_bytes":210,"redis_shared_state_calls":18,"redis_shared_state_duration_s":0.150542,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":1502,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.009,"db_main_duration_s":0.009,"db_main_replica_duration_s":0.0,"cpu_s":0.006104,"mem_objects":3187,"mem_bytes":847616,"mem_mallocs":572,"mem_total_bytes":975096,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_bulk_cron_worker.records_count":0,"duration_s":0.170527,"completed_at":"2022-12-20T11:11:05.152Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.009407}
./gitlab/sidekiq/current:{"severity":"INFO","time":"2022-12-20T11:11:06.768Z","message":"Cleaning working queues"}
./gitlab/gitaly/gitaly_ruby_json.log:{"type":"gitaly-ruby","grpc.start_time":"2022-12-20T11:11:07Z","grpc.time_ms":0.185,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":3618832,"correlation_id":"ef66d28a24f3865f3657944957b979df","time":"2022-12-20T11:11:07.776Z"}
./gitlab/gitaly/gitaly_ruby_json.log:{"type":"gitaly-ruby","grpc.start_time":"2022-12-20T11:11:07Z","grpc.time_ms":0.163,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":3618831,"correlation_id":"60f34ba7cc01139d498ca9abe37636b2","time":"2022-12-20T11:11:07.776Z"}
./gitlab/redis/current:2022-12-20_10:11:16.09983 108332:M 20 Dec 2022 11:11:16.099 * 10 changes in 300 seconds. Saving...
./gitlab/redis/current:2022-12-20_10:11:16.10061 108332:M 20 Dec 2022 11:11:16.100 * Background saving started by pid 4110013
./gitlab/redis/current:2022-12-20_10:11:16.24254 4110013:C 20 Dec 2022 11:11:16.242 * DB saved on disk
./gitlab/redis/current:2022-12-20_10:11:16.24336 4110013:C 20 Dec 2022 11:11:16.243 * RDB: 2 MB of memory used by copy-on-write
./gitlab/redis/current:2022-12-20_10:11:16.30125 108332:M 20 Dec 2022 11:11:16.301 * Background saving terminated with success