Deploy fluentd and perform local log parsing and shipping to Elastic Cloud

The simplest and fastest way of solving our immediate problems with logs is to leverage fluentd and Elastic Cloud.

  • fluentd will be deployed with chef-client
  • fluentd will track and parse system-level logs on all nodes (auth logs, syslog logs in general, etc)
  • fluentd will also track and parse application-specific logs (gitlab-shell, postgresql, redis, etc)
  • fluentd will have a modular configuration approach so we can choose what logs to track, and where, based on chef roles
  • fluentd will write logs directly to the Elastic Cloud cluster via HTTPS + authentication

In order to achieve this, the steps required are the following:

  • URGENT logstash node cannot cope with all the current log parsing load gitlab-cookbooks/gitlab-elk!93 (merged)
  • #164 (closed) Create a gitlab_fluentd cookbook
  • #164 (closed) Make the logging configuration modular (each role will track different log files)
  • Create staging cluster in Elastic Cloud
  • Handle Elastic Cloud credentials with chef-vault in staging
  • #167 (closed) Start gathering system logs in staging
  • #167 (closed) Start gathering gitlab logs in staging
  • #167 (closed) Start gathering haproxy logs in staging
  • Create production cluster in Elastic Cloud
  • Handle Elastic Cloud credentials with chef-vault in production
  • Start gathering system logs in production
  • Start gathering gitlab logs in production
  • Start gathering haproxy logs in production
  • Double check for the need of which applications we need to gather logs from

/cc @andrewn @jarv

Edited by Ilya Frolov
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information