Customization of the fluentd deaemon for cloud logging

As we start to process more logs from the application in Kubernetes we will need to start processing them like we do now for td-agent. Unfortunately GKE doesn't offer a way to configure basic cloud logging, which is something we would need to do for

• Adding additional tags, environment, type, tier, etc.
• Redacting of tokens

To do this we need to deploy the fluentd daemonset to the GKE cluster as documented here https://cloud.google.com/solutions/customizing-stackdriver-logs-fluentd

this means that we will no longer be using the generic gke index for application logs, intead we will forward logs to existing indexes.

• Decide if this is something that we want to support upstream in the gitlab chart: We will not include this in the chart but install fluentd in a separate namespace, since we are likely using an instance pod.
• Figure out how we are going to manage the configuration split between the fluentd coobkook and kubernetes for fluentd config
• Validate a configmap and daemonset that will replicate our our td-agent configuration (see https://github.com/GoogleCloudPlatform/kubernetes-engine-customize-fluentd/tree/master/kubernetes for example)
• Create a new helm chart for deploying the fluentd daemonset
• Create a a k8s-workloads/logging configuration
• Deploy helm chart to all environments (preprod, staging, production)
• Exclude application logs from the pubsub sink that are being sent directly to elasticsearch