Investigate: Log volume from gitlab-shell
With the migration of gitlab-shell from VM's to Kubernetes, we'll see a change to our logs due to the way our containers capture logging. Utilize this issue to evaluate the log data that we'll observe such that we can better inform ourselves for future improvements to either the container, or our fluentd configuration when sending logs to our elasticsearch cluster.
Acceptance criterion
- Get logging from our Pods into ES
- Evaluate the current log volume in comparison to the new log volume due to the log data we'll receive from our Pods.
Milestones:
-
Investigate how to determine index sizing -
Validate index sizing is not excessively large for the gitlab-shell after traffic is being taken in by the GKE infra -
Build support into our fluentd helm chart for filtering messages -
Implement rules inside of the fluentd configuration to drop unnecessary log messages from gitlab-shell - unstructured logs - which includes all log data from sshd and output from
tail
- if the above is not possible, match any data from
tail
and remove that <- minimally this
- unstructured logs - which includes all log data from sshd and output from
Edited by John Skarbek