DRI Memory Team ## Background As our usage of Redis grows, we often partition it to support that growth. One of the first partitions is described here: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7199. In July 2021, we completed work to move trace chunks to their own dedicated redis instance. https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/462. This saw a [5% reduction in CPU usage and a 40-50% reduction in network traffic](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/462#note_627163760) from redis-persistent. We are seeing further growth on `redis-persistent` and need to take action before we are able to conclude our research into a longer-term strategy for scaling Redis https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/557. ## Why did we choose to move session keys next? In https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12821 we discussed approaches to take for partitioning `redis-persistent` and determined that the highest impact would be moving either the database load balancing keys, or the sessions keys to their own instance. (see https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12821#note_533452403). The first suggestion was to do the load balancing keys first since this is easier and lower risk. This also seemed like an appropriate fit for a project that would be done outside of the Scalability team. However, with the [recent production incident](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5546) where session information nearly saturated this instance, it seems better to address the session keys first and make additional allowances to accommodate the risk with this rollout. ## Other projects where we have done similar work 1. Redis for Tracechunks : https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/462 1. Redis for Rate Limiting : https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/526 These projects follow a similar pattern and can be helpful when trying to see how instances have been set up in the past. ## Work Required Firstly, analysis is required to determine the expected impact of this work and to set up the techniques required to perform the same analysis again after the migration is complete. Then there are two lines of work: 1. Update the rails application to use the new instance, and add this instance to the helm charts and omnibus. 1. Build the new VMs for this instance. (The change issues should be prepared by the Memory team for execution by a Scalability SRE) Finally, the migration plan should be created and reviewed. Once the discussion is complete on the issue, a change issue can be created with detailed steps to be followed. The migration is completed together with a Scalability SRE. ## Expectation Between Scalability and Memory teams - Scalability to build out a project plan and create issues - Memory to select a DRI for this project - Memory team will work on the project with assistance available from Scalability for pairing, reviews, or any questions concerning this project - Memory to create rollout plan and change issue that will be reviewed by Scalability - Scalability SRE will execute change issue Please use `#scalability-579-redis-for-sessions` for communication on this project. ### Expectation of the Memory DRI Please would the DRI from Memory be responsible for: - updating the status block below once per week on Wednesdays - keeping track of all issues on the project - letting Scalability know if any reviews are taking too long - raising any concerns or potential problems ## Completed Work #### Implementation: - [x] **Step1** - [Build the Redis::Store instance and use it in session_store](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14507) - [x] **Step2** - [Implement MultiStore with fallback mechanism](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14508) (The Multistore talks directly/only to the `SharedState` by default) - [x] **Step3** - [Use Redis::Sessions instead of Redis::SharedState](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1334) #### Follow up work (non blocking): - [x] [Fix logging method_missing commands in Redis::Multistore](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75630) - [x] [Add use_pirmary_store_as_default feature flag for MultiStore](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75258) - [x] [Fix Multistore FF check for empty test DB](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75616) #### Configuration: - [Build Redis VM's for the redis-sessions instance](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1309) - [x] Chef roles - [x] Terraform - [x] Runbooks - [x] Configure the instance in the application - [x] `gstg` - Change issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5962 - [x] `gprd` - Change issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5969 - [x] Expire the [silence on `type="redis-sessions"`](https://alerts.gitlab.net/#/silences/537ed54e-5b20-4b97-9f58-9fe961aa7385) once traffic levels are stable. #### Rollout: - [x] `Gitlab::Redis::SharedState` is now replaced with `Gitlab::Redis::Sessions`. - [x] `ENV['GITLAB_USE_REDIS_SESSIONS_STORE']` is enabled by default - [x] [Enable `use_primary_and_secondary_stores_for_sessions` feature flag](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1429) - [x] Enable FF on `gstg` environment - [x] Enable FF on `gprd` environment - [x] [Monitor `gitlab_redis_multi_store_read_fallback_total` metric for fallbacks](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1446) - [x] [Enable `use_primary_store_as_default_for_sessions` feature flag](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1445) and disable `use_primary_and_secondary_stores_for_sessions` FF, when `gitlab_redis_multi_store_read_fallback_total` doesn't show any more fallback reads. (~ 1 week after we enabled `gitlab_redis_multi_store_read_fallback_total`) ### Status 2022-02-09 With the Redis Session instance is live and stable, [documentation] (https://docs.gitlab.com/ee/development/redis/new_redis_instance.html#proposed-solution-migrate-data-by-using-multistore-with-the-fallback-strategy) updated, and new instance resized, we consider this project successfully done. Measured impact ([more details](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1396#note_815472671)): - We saved about 51% of persisted data storage for main Redis instance ([source](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1396#note_816555277)) - The total rate of commands dropped to 55% of what it was previously ([source](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1396#note_816538973), [operation graph](https://dashboards.gitlab.net/d/redis-main/redis-overview?orgId=1&from=now-90d&to=now&viewPanel=68)) - Network traffic is reduced by ~40% ([network in](https://dashboards.gitlab.net/d/redis-main/redis-overview?orgId=1&from=now-90d&to=now&viewPanel=74), [network out](https://dashboards.gitlab.net/d/redis-main/redis-overview?orgId=1&from=now-90d&to=now&viewPanel=72)) - We reduced memory footprint and CPU usage per node by ~30%. ([RSS memory](https://dashboards.gitlab.net/d/redis-main/redis-overview?orgId=1&from=now-90d&to=now&viewPanel=88), [CPU](https://dashboards.gitlab.net/d/redis-main/redis-overview?orgId=1&from=now-90d&to=now&viewPanel=70)) This resulted in reducing the current instance size (and cost) from: **c2-standard-3030** (30 vCPUs, 120 Gb, $902/mo \*3) to **c2-standard-16** (16 vCPUs, 64 Gb, $481/mo \*3)