DRI: @cmiskell # Background A trace chunk leak lead to excessive Redis storage in https://gitlab.com/gitlab-org/gitlab/-/issues/327263. To avoid memory saturation, the instance was expanded in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4194. This incident highlighted the approach that we currently use, of storing trace chunks in the Redis Persistent, or shared state, instance. Trace chunk storage is a distinctive workflow from much of the other traffic we send to Redis persistent instance. Additionally, it makes sense to isolate this traffic, since the availability of GitLab.com is highly reliant on the functioning of the persistent redis (authentication, sessions, load balancing, more), so moving this traffic to it's own independent instance would make sense. A proposal described here: https://gitlab.com/gitlab-org/gitlab/-/issues/327469 looks to move trace chunk storage to it's own dedicated instance. This was on the backlog for ~"group::continuous integration", but since they have been quite busy contributing to rapid action and infradev issues, we decided to pick up this project. The related issue is closed in favour of this epic. ## Expected Benefit From the comment: https://gitlab.com/gitlab-org/gitlab/-/issues/327469#note_556531587 We expect to see 5% reduction in CPU usage and 50-60& reduction in network traffic on redis-persistent. ## Cleanup - [x] Revisit if compression is required: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13449 - [x] Let ~"team::Reliability" know that the alert thresholds can be changed: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12796 ### Status 2021-07-08 After trace chunks were moved to their own Redis instance we saw a [5% reduction in CPU usage and a 40-50% reduction in network traffic](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/462#note_627163760) from redis-persistent.