Adding JSON logging to Sidekiq and Rails
https://gitlab.com/gitlab-com/support-forum/issues/851#note_13211170 is an example of how having a central dashboard in Kibana where we can monitor the health of Rails, Sidekiq, and other apps is helpful.
The problem today is that most of the Sidekiq and Rails logs are not stored in a format that makes it easy to extract the requisite data out of it. For example, Sidekiq message might produce:
2016-07-21_02:46:46.71232 worker1 sidekiq: 2016-07-21T02:46:46.712Z 42865 TID-av8m4 ElasticIndexerWorker JID-4396f7b6d349a1bd4528f384 INFO: arguments: ["update", "Issue", 2618503, {"changed_fields"=>[]}]
We don't parse what's inside of this message. Creating a grok filter to parse this is error-prone and difficult, if not impossible, to get right.
For GitLab.com, we should consider adding an option to generate JSON logs, such as the approach taken:
- http://www.avitzurel.com/blog/2014/12/15/open-sourcing-the-way-we-centralize-logs-for-our-rails-applications/
- https://github.com/Springest/Sidekiq-Logging-JSON
- http://www.fluentd.org/datasources/rails
Work to do:
-
Add structured logging for Rails processes: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12928 -
Add structured logging for Grape API: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14102 -
Add support for Sidekiq: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18121 -
Move Sidekiq exporter logs to a different file: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18127