Ingest diagnostic reports uploader logs
The problem
For our diagnostic reports uploader, we log everything into log/diagnostic_reports_json.log
under the Rails root: https://gitlab.com/gitlab-org/gitlab/-/blob/master/bin/diagnostic-reports-uploader#L24.
Note that we run diagnostic reports uploader in a separate container under webservice
pod. It is built from the toolbox
image: gitlab-com/gl-infra/k8s-workloads/gitlab-com!2168 (merged)
As discussed in #375260 (comment 1157481385), we don't scrape this file at all, so it's not currently available in Kibana.
We want to change that.
Technical details.
The report daemon already exports JSON logs: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/memory/diagnostic_reports_logger.rb
As suggested by Matthias here, we may want to set up new Fluentd
rule to ingest this file and send it to Kibana.