208,000 WebHookWorker delivery jobs fail every week on GitLab.com with incompatible character encodings: ASCII-8BIT and UTF-8
Related error in a different part of the application: #21768 (closed)
A large number of WebHookWorker
Sidekiq background jobs fail with the error incompatible character encodings: ASCII-8BIT and UTF-8
.
We should resolve this.
ELK Search: https://log.gprd.gitlab.net/goto/7c980d00602de0948e73cb7c1ee883ac
Stacktrace
lib/gitlab/json.rb:248:in `join',
lib/gitlab/json.rb:248:in `encode',
app/services/web_hook_service.rb:88:in `make_request',
app/services/web_hook_service.rb:39:in `execute',
app/workers/web_hook_worker.rb:16:in `perform',
lib/gitlab/metrics/sidekiq_middleware.rb:18:in `block in call',
lib/gitlab/metrics/transaction.rb:61:in `run',
lib/gitlab/metrics/sidekiq_middleware.rb:18:in `call',
lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:32:in `perform',
lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:40:in `perform',
lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call',
lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context',
lib/gitlab/sidekiq_middleware/worker_context/server.rb:16:in `call',
lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call',
lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call',
lib/gitlab/sidekiq_middleware/admin_mode/server.rb:8:in `call',
lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:7:in `call',
lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call',
lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call',
lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call',
lib/gitlab/with_request_store.rb:17:in `enabling_request_store',
lib/gitlab/with_request_store.rb:10:in `with_request_store',
lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call',
lib/gitlab/sidekiq_middleware/server_metrics.rb:35:in `call',
lib/gitlab/sidekiq_middleware/monitor.rb:8:in `block in call',
lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job',
lib/gitlab/sidekiq_middleware/monitor.rb:7:in `call',
lib/gitlab/sidekiq_logging/structured_logger.rb:18:in `call'
Edited by Andrew Newdigate