Trace Chunks: Build Redis VMs

To move the storage of CI job trace chunks from the persistent/shared Redis to a dedicated Redis, we need to build the Redis store.

Basics

VMs are still the preferred option for persistent storage until we choose to make a concerted effort to migrate these to kubernetes (including shared tooling, experience, documentation, etc)

Per our existing Redis deployments, we will need a 3 VMs "cluster" with Redis Sentinel for failover. While we possibly 'should' use a single shared sentinel for all our clusters (see https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11389) we will run sentinel for this cluster on the Redis nodes as it's pretty much a wash in terms of failure resilience, is consistent with existing deployments, and doesn't bring in additional unnecessary complexity.

Sizing

c2-standard-4 (c2-standard-4 with 4 CPUs and 16GB RAM) seems likely to be more than sufficient. We definitely want a C2 type (the CPU is known to be our best option for Redis right now). Traces are transient under all normal circumstances, and the instigating incident can reasonably be considered highly anomalous, unlikely to be repeated, and managed by scaling up the nodes in response if absolutely necessary. By bytes, the 'string' storage in our redis-persistent is around 7GB (and that includes all strings, not only transient traces), so a 16GB instance could replace all of that with room to spare. We could consider N2's instead so we could customize (reduce) the RAM allocation, but that enters lesser known territory and the risk may not be worth the fairly minimal savings (in the order of a $400/year to remove 8GB RAM from 3 nodes).

Naming

Initial options:

  1. redis-traces
  2. redis-trace
  3. redis-tracechunks
  4. redis-citraces

Personally I prefer redis-tracechunks because redis-traces and redis-trace implies tracing of redis, and citraces is just a mouthful and looks weird, and I wouldn't want to fix that with another hyphen like (redis-ci-traces).