Skip to content

fix: fluent bit output format compatible with newer versions of OpenSearch

Maxim Beder requested to merge maxim/fix-fluent-bit-logs-upload-to-opensearch into main

Description

After upgrading one of the grove clusters, which included bumping up the version of the OpenSearch helm chart, the fluent bit stopped uploading logs to OpenSearch. The errors indicated that OpenSearch considered the format that fluent bit was outputing invalid. After a quick google search, we came up with the following change, that fixes the issue.

Supporting information

Example error logs that were discovered on the fluent bit pods:

[2024/03/21 13:18:21] [error] [output:opensearch:opensearch.0] HTTP status=400 URI=/_bulk, response:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

Testing instructions

Tested on a client's cluster. To apply the fix, run ./tf plan && ./tf apply, check that the fluent bit pods have been restarted with the new config, check their logs - they shouldn't have the errors any more, and check the OpenSearch dashboard for the new data.

Checklist

If any of the items below is not applicable, do not remove them, but put a check in it.

  • All providers include the new feature/change
  • All affected providers can provision new clusters
  • Unit tests are added/updated
  • Documentation is added/updated
  • The TOOLS_CONTAINER_IMAGE_VERSION in ci_vars.yml is updated
  • The grove-template repository is updated

Merge request reports