Discussion: Long term view on Redis (deployment modes and upgrades)
From gitlab-com/gl-infra/k8s-workloads/tanka-deployments!908 (comment 1918162983)
Maintenance toil
We have 2 Redis instances running on GKE: registry cache and pubsub. We are behind on bitnami chart upgrades but these chart upgrades often comes along with a new Redis default version. For past upgrades, we ignore it by pinning to Redis 6.2.x
, reviewing the diff and testing on pre/gstg first.
Performance
Recently, we have looked into with the performance issues of packet processing in Kubernetes
Discussion scope
Some questions @igorwwwwwwwwwwwwwwwwwwww brought up which we should decide on to facilitate Scalability group's longer term view on Redis (affects scaling, maintenance, etc)
- Do we still want to move all of our Redis into Kubernetes long-term?
- If yes, can we define a proper upgrade strategy and procedure for it (with ability to fall back to a previous RDB)?
- If no, should we migrate the existing Redises in Kubernetes back out into VMs?
We have operated Redis in GKE for non-trivial workload (pubsub) and Redis on VMs (gitlab-redis cookbook, not omnibus installations) too. I think we do have enough data points to make a decision compared to previous attempts in the past where we effectively were stuck at an impasse.