Sidekiq emits unstructured logs
In an effort to surface unstructured logs, we've discovered some being emitted by sidekiq: https://log.gprd.gitlab.net/goto/2f110297c1cdc8d849e4b1fe054de658.
There's a few that I can see.
startup message:
2020-03-04T10:56:39.953Z 16689 TID-ot8v3b7h0 INFO: Starting cluster with 8 processes
...
2020-03-04T10:57:05.500Z 16709 TID-oter1foxh INFO: Booting Sidekiq 5.2.7 with redis options {:scheme=>"redis", :host=>"gprd-redis-sidekiq", :port=>6379, :password=>"REDACTED", :id=>nil, :sentinels=>[{:host=>"redis-sidekiq-01-db-gprd.c.gitlab-production.internal", :port=>26379}, {:host=>"redis-sidekiq-02-db-gprd.c.gitlab-production.internal", :port=>26379}, {:host=>"redis-sidekiq-03-db-gprd.c.gitlab-production.internal", :port=>26379}], :namespace=>"resque:gitlab", :url=>nil}
Thread crash:
#<Thread:0x00007ff23ecbef98@job_monitor@/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/daemon.rb:42 run> terminated with exception (report_on_exception is true):
(stacktrace)
Some tickets related to structured logging with sidekiq: gitlab-foss#20060 (closed), gitlab-foss!18121 (merged). /cc @stanhu @smcgivern
It's also possible that separating structured logs out and changing the way we ingest them on gitlab-com is what's needed here.
refs https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9274