## DRI @nduff ## Participants @poneill ## Problem Statement Currently, we're using Fluentd to send logs to Elastic and our archives in GCS. These have been having several issues around memory not being collected by the garbage collector and causing memory usage to slowly grow over time. We have tried several methods to tune the garbage collector, all without success. So far, restarting the pods periodically has been the best method for temporarily fixing the problem. See here https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2003 and here https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2004 for more information. Fluentd has also caused numerous non-user facing incidents that page the on call, causing a significant burden on them, and a maintenance burden on us. The latest incidents were: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19749 & https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19733 We have already determined [vector.dev](https://vector.dev) will be a part of the logging solution we're building so it makes sense to switch over instead of continuing to spend time trying to fix the issues we're having with Fluentd. See here https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1430 and here for more information https://gitlab.com/gitlab-com/gl-infra/observability/team/-/issues/4119. We have also discovered a significant cost savings opportunity with the Security cluster that raises the priority of this work. See https://gitlab.com/gitlab-com/gl-infra/observability/team/-/issues/4333 for more details. ## Exit Criteria - [x] Replace `fluentd-elasticsearch` with `vector` in all gitlab.com environments. - [x] Replace `fluentd-archiver` with `vector` in all gitlab.com environments. <!-- STATUS NOTE START --> ## Status 2025-11-26 :clock1: **total hours spent this week by all contributors**: 18 hours :tada: **achievements**: A new [vector-config](https://gitlab.com/gitlab-com/gl-infra/charts/-/tree/165345a6287700dd3824ac14fc62989db8fbb523/gitlab/vector-config) chart has been created to help move our existing jsonnet config generation into helm. It provides the ability to create re-usable templates to use across services/pipelines. This now unblocks our progress to move on to our necessary rollout steps. :arrow_forward: **next**: Two new sub epics have been created to break down the rollout tasks required for replacing `pubsubbeat` and `fluentd-elasticsearch`. With the creation of our config chart we will now move onto these and being to start replacing fluentd! _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1551#note_2913989805_ <!-- STATUS NOTE END -->