Skip to content

Refactor fluentd logging-config.yaml to Jsonnet for better maintainability and reduced duplication

The fluentd logging configuration in lib/fluentd/logging-config.yaml currently contains significant duplication and lacks clarity in its configuration patterns. This issue was highlighted by a recent change in MR !1269.

The current YAML configuration requires manual duplication of similar log forwarding rules across:

  • Multiple services (web, api, internal-api, git, websockets, ai-assisted, etc.)
  • Multiple stages (main, cny)
  • Multiple components (workhorse, rails, sidekiq shards)

This results in ~1000+ lines of repetitive configuration that is error-prone to maintain and difficult to understand for engineers unfamiliar with the codebase.

Examples of duplication patterns:

  1. Service-stage combinations: Each service (web, api, git, etc.) is duplicated for main/cny stages
  2. Component types: Rails and Workhorse configurations follow identical patterns with only minor differences
  3. Sidekiq shards: 15+ sidekiq shard configurations share the same structure
  4. Common fields: Fields like elastic_key_prefix, keep_time_key, and logstashPrefix are repeated identically

Proposed Solution

Convert lib/fluentd/logging-config.yaml to Jsonnet to leverage:

  1. Functions and abstractions: Create helper functions for common patterns (e.g., createWebserviceConfig(serviceName, stage))
  2. Default values: Define defaults once and override only when necessary
  3. Type safety: Leverage Jsonnet's type system to catch configuration errors
  4. Reduced cognitive load: Engineers only need to specify what's different, not repeat everything
  5. Better documentation: Inline documentation in Jsonnet with clear examples

Benefits

  • Maintainability: Changes to common patterns only need to be made once
  • Clarity: Configuration intent is clearer when defaults are separated from overrides
  • Consistency: Enforced patterns reduce configuration drift
  • Reduced errors: Less duplication means fewer opportunities for copy-paste mistakes
  • Better onboarding: New engineers can understand the configuration structure more quickly

Acceptance Criteria

  • Convert YAML configuration to Jsonnet
  • Create helper functions for common configuration patterns
  • Document the new Jsonnet structure with examples
  • Ensure generated configuration matches existing YAML output (no functional changes)
  • Update contributing documentation to explain the new Jsonnet approach
  • Verify all environments compile successfully with the new configuration