Evaluate feasibility of hybrid deployment during Redis migration
In order to migrate safely, we'd ideally be able to run in a hybrid deployment where some redis replicas are in VMs, and some are in Kubernetes.
This would enable us to:
- Directly compare the performance and behaviour.
- Migrate data into Kubernetes.
- Failover to a primary in Kubernetes.
- Fail back to VM if needed.
- Roll back to a fully VM based deployment if needed.
However, the bitnami chart as well as our current deployment may not support this setup out of the box.
The purpose of this issue is to evaluate if such a hybrid setup is at all feasible, and if so, produce a proof of concept.
Testing
- Wiring up VM's and Kubernetes together: #1468 (comment 794221159)
- Network Partitioning: #1468 (comment 806265830)
- Failovers:
- Via Sentinel: #1468 (comment 804715384)
- A bit more testing here: #1468 (comment 804665906)
- Disk failure of a Pod: #1468 (comment 803694056)
- Migration procedure (linking Kubernetes and VM's together): #1468 (comment 808788309)
Other observations seen during this testing: #1468 (comment 804684443)
Milestones
-
setup redis with sentinel cluster inside of preprod -
wire up gitlab in preprod to utilize this redis -
test migration procedures -
link VM redis with Kubernetes Redis -
determine failure modes during this configuration -
test fail overs -
test network latencies between app and redis -
test data loss scenarios -
...
-
-
... -
Clean this all up when we are done
Edited by John Skarbek