Consolidate log shippers in kubernetes

Problem

We currently run two log shipper daemonsets in Kubernetes:

There are a few suboptimal consequences of this:

  • Since the GKE fluent bit configuration reads all container log files, we're reading the highest-throughput log files (our own apps) twice. As we migrate more workloads to kubernetes, this will become more painful.
  • Our own apps' logs are stored in stackdriver (as well as ES), which incurs some costs
    • I have not quantified this cost yet
  • Confusion for SREs: some log shipping config lives in our own Kubernetes config, and some is not version controlled (by us) and is part of the GKE platform).

Desired outcome

All log shipping/pipeline configuration is version controlled in one place, presumably our fluentd configuration that we currently store in https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/tree/master/releases/fluentd-elasticsearch (unless it has been moved to tanka).

Acceptance criteria

  • All log shipping/pipeline configuration is version controlled in one place.
  • Log files are only read once in Kubernetes, and any required fanning out / routing logic is part of the log pipeline.