## Background Rack::Attack uses `Rails.cache` by default as it's backend. For us this means redis-cache. As noted by @jacobvosmaer-gitlab in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3034#note_460538394, there was a noticeable increase in CPU usage on the cache instance when we started tracking the rate limiting (dry run). With CPU saturation on the redis-cache instance reaching ~85% at peak times, it's time to consider splitting this out to it's own Redis. `Rack::Attack` allows this to be configured as a `ActiveSupport::Cache`: https://github.com/rack/rack-attack#cache-store-configuration ## Expected Benefit The [graph](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3034#note_460538394) indicated a raise from CPU saturation peaks of 50% to peaks of almost 70%; if the contribution to total saturation has remained proportionate, this now accounts for 24% (absolute) of the 85% peak, and splitting this out would reduce saturation peaks to around 60%. If the contribution remains literal/absolute (20%), then the reduction will be to around 65%. Either is good, and would give a lot of breathing room on redis-cache. ## Steps 1. [x] Build new a set of Redis VMs for RackAttack (`redis-ratelimiting` as the prime identifier) 1. [x] Add capability to the application to use a distinct Redis backend 1. [x] Add configurability of this new Redis to the helm charts and omnibus 1. [x] Switch to the new store Notes: 1. Switching over should be done quickly, but does not need to be instantaneous across the fleet because the rate-limiting windows we use are only 1 minute. If there was a few 10s of minutes where processes across the fleet are split in which cache they use, the rate-limit for any given user/IP will be *potentially* doubled although that's the worst case, and is subject to how requests are randomly distributed. We still have haproxy providing a front-end per-IP rate-limit as well. Doing this at a normally quiet time would be quite acceptable. A feature flag would be nice, but isn't mandatory. If a feature flag is possible, it needs to be global; it cannot be "% of requests" (for correctness) or "% of actors" (at best we know the user not project, and need to handle unauthenticated traffic as well). 1. With that in mind, I think we're going to have to treat the configuration as an initialization time thing. While the store can be changed on the fly, doing so [instantiates](https://github.com/rack/rack-attack/blob/master/lib/rack/attack/cache.rb#L19) a new proxy object for a [redis](https://github.com/rack/rack-attack/blob/master/lib/rack/attack/store_proxy/redis_proxy.rb) connection which we really don't want to do on every request. To make it feature-flag compatible we'd instead have to check if the feature-flag had *changed* during every request (or periodically) and only reconfigure the store when it does. All of which is complex and potentially unnecessary given having it be configured only at init time is tolerable. ## Status 2021-10-13 This is complete. We've observed CPU saturation on the redis-cache primary drop from ~90% to ~70% due to this work, with no reported user impact.