Build Redis VM's for the redis-ratelimiting instance
To move the RackAttack storage from the Cache Redis to a dedicated Redis, we need to build the Redis store.
Basics
VMs are still the preferred option for persistent storage until we choose to make a concerted effort to migrate these to kubernetes (including shared tooling, experience, documentation, etc)
Per our existing Redis deployments, we will need a 3 VMs "cluster" with Redis Sentinel for failover. As for tracechunks we will run sentinel for this cluster on the Redis nodes as it's pretty much a wash in terms of failure resilience, is consistent with existing deployments, and doesn't bring in additional unnecessary complexity.
Sizing
The current cache nodes are c2-standard-30 with 30 CPUs and 120GB RAM, exclusively for the RAM (even with I/O threading, load only peaks at about 1.2CPU)
So how much memory does RackAttack use? We can scan the keyspace using redis-cli matching a pattern (or via Ruby) but that takes a few minutes and the keys only have meaning for one minute so we don't get a sufficiently consistent/limited view. Instead, we can monitor traffic for 60 seconds and extract the keys that are touched in that time and count the number of them. At 04:30 UTC on a Monday (low usage time) we see 32K unique combinations passing in a 60 second period.
Another way is count the unique users and IP addresses seen by Rails in any given minute (each results in a key in Redis). We can get these from logs in Kibana (https://log.gprd.gitlab.net/goto/fd6a2c606b7b96e3d4b0a68dadad2804), and this correlates well with the measurement using traffic monitoring (in the order of 30-33 thousand), which means we can use that same query to see what happens at peak times without actually capturing raw data at that time. That says we see peaks of just around 65K under normal circumstances.
Values are an integer, but stored as a string, and with our rate-limit values these could be up to 4 digits long (4 bytes), but there's overhead for storing a key + string, and memory usage <key> reports 120 bytes due to the key name (of the form cache:gitlab:rack::attack:<number>:throttle_authenticated_web:<number>/<IPaddress>, which are 70-80 chars).
So we need 65K*120 = or just under 8MB, so I'm going to go ahead and suggest we don't need a large amount of RAM here and it's much more about the CPU.
We also have Application Limits (added to the project after the above was done). Storage per key is around 100 bytes (empirical inspection of a few) it's much harder to turn logs into estimates because the rate-limits often apply at lower levels (in /lib) making endpoint usage counting not terribly accurate. So we're going to have to stick to inspecting passing traffic in Redis. Over one minute at 23:50 UTC on a Tuesday, 1470 keys were used; if there were an order of magnitude higher usage at peak times (unlikely), that's still only in the order of 1.5MB.
c2-standard-4 (c2-standard-4 with 4 CPUs and 16GB RAM) is the smallest C2 (our preferred Redis CPU), and will be more than sufficient RAM wise; we will monitor directly when we migrate to ensure the math holds up, but unless I'm out by 3 orders of magnitude, we're good.
Scaling (memory)
To put a quantity on how much headroom this memory usage gives us:
Assuming we don't want to hit more than about 50% RAM to allow for worst-case-scenario with the fork for background save, and noting that the 1-minute rack-attack rate-limiting window works heavily against us here (if the bgsave takes more than (on average) 1 minute then we will indeed need double the RAM because the dataset will cycle in its entirety), we have 8GB to play with. At 120 bytes per entry for RackAttack, 8GB allows around 71M unique IP addresses generating traffic in any minute. Our current peak request/minute is around 780K across all IPs and per rates noted above that's an average of a little over 10 requests per IP in any minute (the distribution will be quite wide though). So 8GB would allow us to handle in the order of 1000x current traffic; CPU will melt long before that (require further division and/or horizontal scaling with something like Redis Cluster).
For Application Rate-Limiting we have scalability around 58000x, if RackAttack didn't grow also (which it would have to), but again, we'd have other problems long before then. Use of the Application Rate limiter may well grow as more specific rate-limits are added, but we'd have to add a huge number of such rate-limits to be a problem.
In short: we have huge amounts of headroom, no concerns.
Naming
redis-ratelimiting - It's the only logical choice (Change My Mind)
History
See #1055 (closed) for recent near-identical activity.