Kubernetes Pod Logs are not displayed in the UI when mapping a Kubernetes Deployment to an Environment

Summary

Kubernetes Pod Logs are not displayed within Operations - Logs when mapping a Kubernetes Deployment to an Environment.

Steps to reproduce

1. Add Kubernetes Cluster into your project or group
2. Install Elasticsearch integration using the UI
3. Deploying a Kubernetes via pipelines. Make sure to add the needed annotations to map the Deployment to an environment (example: https://gitlab.com/gitlab-commit-demo/ops-sample/-/blob/master/chart/templates/deployment.yaml)
4. Review the logs (Operation - Logs)

Example Project

Sample project: https://gitlab.com/gitlab-commit-demo/ops-sample/-/blob/master/chart/templates/deployment.yaml

What is the current bug behavior?

Kubernetes Pod logs are not displayed in the UI (Operation - Logs)

The root cause of the issue is because of the needed annotations to map the deployment to an environment (https://docs.gitlab.com/ee/user/project/deploy_boards.html#enabling-deploy-boards). Those annotations containing a dot that breaks Filebeat which then ends up in logs not being delivered to Elasticsearch.

What is the expected correct behavior?

Kubernetes Pod should be displayed in the UI (Operation - Logs)

Logs with the needed annotations should be delivered to Elasticsearch via Filebeat.

Relevant logs and/or screenshots

Filebeat output:

failed to parse field [kubernetes.labels.app] of type [keyword] in document with id 'WyKuZHEBvBLkRerLfpPA'

elasticsearch/client.go:539	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x28ce3758, ext:63722128887, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"source":"/var/lib/docker/containers/XX/XX-json.log", "message":"      Executed endpoint 'sample_mvc.Controllers.HomeController.Index (sample-mvc)'", "prospector":common.MapStr{"type":"docker"}, "input":common.MapStr{"type":"docker"}, "beat":common.MapStr{"name":"elastic-stack-filebeat-f9prn", "hostname":"elastic-stack-filebeat-f9prn", "version":"6.7.0"}, "meta":common.MapStr{"cloud":common.MapStr{"instance_id":"61cd1175-0c8e-4576-9e33-803425d7f0b1", "instance_name":"XX", "machine_type":"Standard_B4ms", "region":"westeurope", "provider":"az"}}, "offset":1442969, "log":common.MapStr{"file":common.MapStr{"path":"/var/lib/docker/containers/XX/XX-json.log"}}, "stream":"stdout", "json":common.MapStr{}, "kubernetes":common.MapStr{"container":common.MapStr{"name":"sample-mvc"}, "namespace":"ops-sample-18054005-production", "replicaset":common.MapStr{"name":"production-app-5d8d45489c"}, "labels":common.MapStr{"pod-template-hash":"5d8d45489c", "app":common.MapStr{"kubernetes":common.MapStr{"io/instance":"sample-mvc", "io/name":"sample-mvc"}}}, "pod":common.MapStr{"uid":"b6b3298e-f297-42a4-8c79-b55a68c3ea07", "name":"production-app-5d8d45489c-mlrnr"}, "node":common.MapStr{"name":"aks-default-41303111-vmss000002"}}, "host":common.MapStr{"name":"elastic-stack-filebeat-f9prn"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42088b6c0), Source:"/var/lib/docker/containers/XX/XX-json.log", Offset:1443122, Timestamp:time.Time{wall:0xbf9c41871cb49227, ext:971583977, loc:(*time.Location)(0x210b240)}, TTL:-1, Type:"docker", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x46e0f9, Device:0x801}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [kubernetes.labels.app] of type [keyword] in document with id 'WyKuZHEBvBLkRerLfpPA'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1285"}}

Output of checks

This bug happens on GitLab.com

Possible fixes

The issue needs to be fixed in the Filebeat configuration which is stored in the elastic-stack-filebeat in the gitlab-managed-apps namespace.

The deployed config:

    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false
      prospectors:
        path: ${path.config}/prospectors.d/*.yml
        reload.enabled: false
    filebeat.prospectors:
    - enabled: true
      paths:
      - /var/log/*.log
      - /var/log/messages
      - /var/log/syslog
      type: log
    - containers.ids:
      - '*'
      json.ignore_decoding_error: true
      json.keys_under_root: true
      processors:
      - add_kubernetes_metadata: null
      - drop_event:
          when:
            equals:
              kubernetes.container.name: filebeat
      - decode_json_fields:
          fields:
          - message
          when:
            equals:
              kubernetes.container.name: modsecurity-log
      type: docker
    http.enabled: true
    http.port: 5066
    output.elasticsearch:
      enabled: true
      hosts:
      - http://elastic-stack-elasticsearch-client:9200
    output.file:
      filename: filebeat
      number_of_files: 5
      path: /usr/share/filebeat/data
      rotate_every_kb: 10000
    output.file.enabled: false
    processors:
    - add_cloud_metadata: null

Adding the two *.dedot=true lines will fix the issue and allows Filebeat to forward the logs to Elasticsearch. After changing the secret all Filebeat Pods need to restarted.

      processors:
      - add_kubernetes_metadata:
          labels.dedot: true
          annotations.dedot: true