Kubernetes Observability: Logging

Overview

This issue tracks the implementation of logs collection and integration for the Runway GKE clusters.

Background

We want to on-board the first pilot customer to Runway on Kubernetes. This requires a base-level of productionization, including observability. This issue is for logs, metrics are tracked in #449 (closed).

Other GKE clusters use fluentd to gather and process logs.

Objectives

  1. Implement logs collection from GKE
  2. Integrate with our existing Kibana/Elastic Search stack

Implementation Details

Note

The implementation outline was generated by Claude. Proceed with caution and sanity check the steps.

  1. Set up GCP resources via Terraform

    • Add Terraform configuration in gitlab-com/gl-infra/platform/runway/provisioner to:
      • Create GCP service account for logging
      • Configure Workload Identity Federation for the service account
      • Set up appropriate IAM policies for log access
      • Apply Terraform changes and verify resource creation

  2. Deploy Vector via Helm and Flux

    • Add logging namespace definition to the Flux configuration in gitlab-com/gl-infra/k8s-mgmt/clusters
    • Add HelmRepository resource pointing to the Vector Helm chart repository
    • Create a Kubernetes Secret or patch configuration for Workload Identity Federation annotations to be applied to the ServiceAccount created by the Helm chart
    • Create supplementary RBAC resources if needed beyond what the Helm chart provides
    • Commit and push changes to trigger Flux reconciliation

  3. Platform log collection

    • Configure Vector Helm values for kubelet logs collection
    • Configure Vector Helm values for kube-proxy logs collection
    • Configure Vector Helm values for container runtime logs
    • Configure Vector Helm values for node system logs
    • Add GKE-specific control plane logs collection configurations

  4. Application log collection

    • Add application-specific parsing rules to Helm values
    • Configure log format standardization
    • Set up multi-line log handling for stack traces
    • Configure application-specific metadata tagging

  5. ClickHouse Cloud integration

    • Set up ExternalSecret resource to fetch ClickHouse Cloud credentials from Vault
    • Configure Vector outputs to send logs to ClickHouse Cloud
    • Set up appropriate schema/tables in ClickHouse (if required)
    • Configure index lifecycle management policies
    • Implement retry and buffering settings for log delivery

  6. Testing and validation

    • Verify Flux is correctly reconciling the HelmRelease
    • Validate Workload Identity Federation is working correctly
    • Verify platform logs appear in ClickHouse
    • Verify application logs appear in ClickHouse
    • Test log rotation and retention
    • Perform load testing to validate performance
    • Document any log format requirements for applications
    • Verify logs are accessible in Grafana (coordinate with Observability team)

  7. Documentation

    • Document the logging architecture with Flux, Helm and Vector
    • Document Workload Identity Federation setup
    • Document ClickHouse Cloud integration and schema design
    • Create runbooks for common logging issues
    • Add monitoring for the logging components themselves
    • Document the GitOps workflow for making changes to the logging configuration

Definition of Done

  • Platform logs are collected by fluentd
  • Application logs are collected by fluentd
  • Platform and application logs are available on https://log.gprd.gitlab.net/
Edited by Florian Forster