Implement dual write strategy in the application
Implement client-side support for the dual-write strategy for migrating redis-cache
.
This migration strategy gives 2 major benefits:
- Lets us pre-warm the cluster (to avoid overwhelming the systems supported by the cache).
- Lets us enable and revert quickly, using feature flags rather than an entire deployment cycle.
This requires adding client-side implementation to support:
- New configuration to specify a second redis-cache service.
- Feature flags to support the following state transitions for the rollout/rollback:
- Use old cache exclusively (initial state): Read and write only to the old cache. Ignore connection failures to the unused new cache.
- Use old cache and actively populate new cache: Write to both caches, but read only from the old cache. Ignore write failures to the unused new cache.
- Use new cache but support rollback: Write to both caches, but read only from the new cache. Ignore write failures to the unused old cache.
- Use new cache exclusively: Write and read only to the new cache. Ignore connection failures to the unused old cache.
When implementing the feature flags to support dual-writes, we have 4 states to support. If we implement those states with a combination of multiple feature flags (e.g. 2 flags, one for reads and one for writes), then we should make it a boolean type of flag, not a percent-of-actors.
See also: GitLab docs: Add a new Redis instance.
Actionables
-
Add and integrate Gitlab::Redis::ClusterCache
with feature flags -
Update MultiStore to prioritise writes to default store for better guarantees -
Implement read-only pipelines for MultiStore to avoid performance regression for cache reads
Edited by Sylvester Chin