Create dedicated Elastic index for User Experience SLIs logs

Previous work in #4288 (closed) successfully set up log collection for User Experience SLIs, but the logs are currently being ingested into existing Rails and Sidekiq indexes rather than a dedicated index:

• pubsub-rails-inf-gprd-*
• pubsub-sidekiq-inf-gprd-*

Desired locations:

• pubsub-user-experience-sli-inf-gprd-*
• pubsub-user-experience-sli-inf-gstg-*

As User Experience SLIs grows in usage and expands to cover more services beyond Rails and Sidekiq, we need a dedicated index to provide a unified view of all User Experiences logs across different services.

Proposed Solution

Based on @nduff's analysis, we have a few potential approaches:

1. Fluentd Configuration Enhancement: Update the current fluentd setup to support filtering by subcomponent: user-experience-slis and routing to a dedicated index. This would involve:

   • Adding regex filter support to include only user-experience-slis logs
   • Updating existing configs to exclude user-experience-slis from rails/sidekiq indexes
   • Ensuring different POS files to avoid position tracking conflicts

2. Vector Migration: Evaluate if migrating to Vector (which we're already considering for some workloads) would be a better fit for this type of log routing customization.

Tasks

• Add the ILM policy for the user-experience-sli index
• Add index patterns to Kibana
• Decide between enhancing fluentd configuration vs. migrating to Vector
• Implement the chosen solution to route user-experience-slis logs to dedicated index
• Verify logs are properly flowing to the new dedicated index