eMail send 4 times - sidekiq issue?

Dear, after updating to 15.10.2 from 15.9.3 we have an issue with sending emails multiple times (4 times) every time.

In the sidekiq error log we see the following (below the complete log line): "error_message":"Net::ReadTimeout with #\\u003cSocket:(closed)\\u003e","error_class":"Net::ReadTimeout",

All services are running fine and restarted all services as well if we do a: gitlab-rake gitlab:check We don't see any errors

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.18.0 ? ... OK (14.18.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Complete log line:

{"severity":"WARN","time":"2023-04-12T10:08:02.581Z","retry":3,"queue":"mailers","class":"ActionMailer::MailDeliveryJob","args":["[FILTERED]"],"jid":"3da60b8905998cfdb7c925b8","created_at":"2023-04-12T10:06:12.145Z","correlation_id":"7e9fc1f3c9def975e4c8189467ddc7f9","meta.caller_id":"NewNoteWorker","meta.remote_ip":"XXX","meta.feature_category":"team_planning","meta.user":"XXX","meta.user_id":18,"meta.project":"XXX","meta.root_namespace":"XXX","meta.client_id":"XXX","meta.root_caller_id":"Projects::NotesController#create","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:mailers:e0ccbdb80f8ada5c8201633e046810f3f4b33f2b3bda1efc2a5a9febacdc6d51","size_limiter":"validated","enqueued_at":"2023-04-12T10:07:56.631Z","error_message":"Net::ReadTimeout with #\u003cSocket:(closed)\u003e","error_class":"Net::ReadTimeout","failed_at":"2023-04-12T10:06:24.429Z","retry_count":2,"retried_at":"2023-04-12T10:07:21.189Z","job_size_bytes":385,"pid":12224,"message":"ActionMailer::MailDeliveryJob JID-3da60b8905998cfdb7c925b8: fail: 5.946594 sec","job_status":"fail","scheduling_latency_s":0.001906,"redis_calls":2,"redis_duration_s":0.000655,"redis_read_bytes":1068,"redis_write_bytes":168,"redis_cache_calls":1,"redis_cache_duration_s":0.000411,"redis_cache_read_bytes":1067,"redis_cache_write_bytes":45,"redis_queues_calls":1,"redis_queues_duration_s":0.000244,"redis_queues_read_bytes":1,"redis_queues_write_bytes":123,"db_count":40,"db_write_count":3,"db_cached_count":9,"db_replica_count":0,"db_primary_count":40,"db_main_count":40,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":9,"db_main_cached_count":9,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.032,"db_main_duration_s":0.032,"db_main_replica_duration_s":0.0,"cpu_s":0.242682,"mem_objects":110684,"mem_bytes":15298520,"mem_mallocs":127075,"mem_total_bytes":19725880,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":5.946594,"completed_at":"2023-04-12T10:08:02.579Z","load_balancing_strategy":"primary","exception.class":"Net::ReadTimeout","exception.message":"Net::ReadTimeout with #\u003cSocket:(closed)\u003e","exception.backtrace":["lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in call'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in call'","lib/gitlab/sidekiq_middleware/worker_context.rb:9:in wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in block in call'","lib/gitlab/application_context.rb:115:in block in use'","lib/gitlab/application_context.rb:115:in use'","lib/gitlab/application_context.rb:55:in with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:in call'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in block in call'","lib/gitlab/database/query_analyzer.rb:37:in within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in block in call'","lib/gitlab/with_request_store.rb:17:in enabling_request_store'","lib/gitlab/with_request_store.rb:10:in with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in call'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in call'"],"db_duration_s":0.029248}`