### Summary We are saturating on cpu for ~"Service::RedisSessions" ([saturation forecast](https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2015)) and hence looking into migrating the workload into a redis cluster for horizontal scalability. ### Participants - BE: @marcogreg - SRE: @fshabir ### Work items 1. Evaluate `RedisSessions` compatibility with Redis Cluster 2. Create configuration changes required for migrating the workload 3. Provision and configure Redis clusters in `pre`, `gstg` and `gprd for `Sessions` 4. Run and verify migration in each of the environments <!-- STATUS NOTE START --> ## Status 2025-03-18 We've finished the post migration cleanups. This epic can be closed during Grand Review :tada: ### Closing Status #### The original problem The ~"Service::RedisSessions" was experiencing CPU saturation, as identified in the [S1 saturation forecast](https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2015). This posed a scalability challenge for our infrastructure. To address this, we needed to migrate the workload to a Redis Cluster architecture to enable horizontal scalability. #### Changes Made * [Provisioned](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/32) and configured Redis Clusters in `pre`, `gstg`, and `gprd` environments for Sessions ~"Service::RedisClusterSessions" * [Developed](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/35) a migration strategy with three key MRs: [supporting rollback from `CacheStore` to `RedisStore`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/176876), [enabling rollforward from `RedisStore` to `CacheStore`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/176108), and [adding MultiStore for dual-writing and failover read](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/175735) - ensuring seamless transition with zero downtime for users * [Executed](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19252) a phased migration approach with dual-writing to ensure data integrity * Completed the migration in all environments with minimal disruption * [Performed](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/34) configuration and code cleanup post-migration #### Impact The migration has significantly improved our infrastructure's performance and scalability: * Reduced CPU utilization from \~90% to \~35% during peak hours <table> <tr> <th>Before</th> <th>After</th> </tr> <tr> <td>  [Source](https://dashboards.gitlab.net/d/redis-sessions-main/redis-sessions3a-overview?from=2024-12-05T10:11:43.599Z&orgId=1&timezone=utc&to=2025-03-05T10:11:43.599Z&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&viewPanel=panel-8) </td> <td>  [Source](https://dashboards.gitlab.net/d/redis-cluster-sessions-main/redis-cluster-sessions3a-overview?orgId=1&from=2025-02-17T13:22:44.406Z&to=2025-03-19T13:22:44.406Z&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-shard=$__all&viewPanel=panel-8) </td> </tr> </table> * Future capacity expansion on Redis Cluster Sessions is now significantly simplified, as Redis Cluster's horizontal scaling architecture allows us to seamlessly add nodes to accommodate growing workloads without complex migration efforts. #### Acknowledgments :handshake: :collaboration: Special thanks to: * @fshabir for the excellent collaboration, taking care of all the infrastructure changes required, performing and pairing through countless of Change Requests while ensuring seamless migration. * @reprazent @schin1(who has sadly left us) for their expertise and collaboration to develop the [multi-phased migration strategy](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/35), ensuring we had a clear rollback/rollforward path in each deployment. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/9#note_2401897184_ <!-- STATUS NOTE END -->