Skip to content

Logging from zoekt-webserver processes forwarded to Elasticsearch

The zoekt-webserver already supports log files with rotation . We should implement the same option in zoekt-dynamic-indexserver.

We should also investigate if it's practical to make these JSON formatted log files and if not we might need a custom parser to deal with sending them in some structured way to Elasticsearch. I assume we already have such tools for many other systems we capture logs from that don't have JSON formatted log files (eg. Postgres).

Infra changes

  1. Create new GCP pubsub topic: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5669
  2. Configure Fluentd to forward logs to GCP pubsub: gitlab-com/gl-infra/k8s-workloads/tanka-deployments!733 (merged)
    1. NOTE: this works on the default non-JSON log parser defined at https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/tanka-deployments/-/blob/7c610c1612af42586bbe278cab53d05f81df490b/lib/fluentd/index.containers.input.conf.libsonnet#L20
  3. Create an index pattern in Elasticsearch: gitlab-com/runbooks!5734 (merged)
  4. Configure pubsubbeat to forward the logs to Elasticsearch: gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!2098 (merged)
  5. Configure Kibana index pattern: https://log.gprd.gitlab.net/app/management/kibana/indexPatterns/patterns/3f6d1a00-e8b6-11ed-a017-0d32180b1390#/
  6. Configure Elasticsearch index template to ensure fields are correctly typed: gitlab-com/runbooks!5739 (merged)
Edited by Dylan Griffith