Add MultiStore support for ActionCable for migration
What does this MR do and why?
This MR introduces a new Redis wrapper for ActionCable that relies on MultiStore.
We are following previous implementations: !126451 (merged) and the existing documentation from: https://docs.gitlab.com/development/redis/new_redis_instance/
MultiStore is used to provide a mechanism to allow for a zero downtime migration (with a minimal gap in which events may not be delivered), and we intend to use this only for gitlab dot com. The code shipped here will be rolled back after migration is successful.
For MultiStore, the secondary_store is the default one where the application runs both read and write operations when no feature flag is enabled. The primary_store is the store we want to migrate to.
In this MR implementation, the secondary_store reads from config/cable.yml while primary_store relies on config/redis.action_cable.yml. Any action cable specific setting is still read from config/cable.yml.
To switch behavior we use the following feature flags:
-
use_primary_and_secondary_stores_for_action_cable(when this is enabled, we publish on both instances) -
use_primary_store_as_default_for_action_cable(when this is enabled we switch the main instance fromsecondary_storetoprimary_store)
References
part of gitlab-com/gl-infra/tenant-scale/tenant-services/team#296
How to set up and validate locally
**NOTE: **
As Redis PubSub doesn't rely on db separation, you need two separate instances (for example, from two gdk installations, or using a docker container).
In order to reduce the noise other parts of the application that relies on Redis (for example, cache, sidekiq and session store), consider setting both config/cable.yml and config/redis.action_cable.yml to different redis instances from what the application is using for everything else.
Test application works without enabling new configuration:
- configure Redis A in
config/cable.yml - start gdk
- open two browser tabs in the same project issue
- make a comment in one tab, see it appears on the second one
Test application works enabling new configuration and not setting feature flags:
- configure Redis B in
config/redis.action_cable.yml - restart gdk
- make another comment in one tab, see it appears on the second one
Enable the first feature flag:
- run
rails console, and inside it:Feature.enable('use_primary_and_secondary_stores_for_action_cable') - open a terminal pointing
redis-clito Redis A (for example:redis-cli -s path_to/gdk_A/redis/redis.socket, orredis-cli -h IP -p PORT_A) - open a terminal pointing
redis-clito Redis B (for example:redis-cli -s path_to/gdk_B/redis/redis.socket, orredis-cli -h IP -p PORT_B) - run
MONITORon both redis clients, to watch for commands being sent to them - make another comment in one tab and see it appears on the second one
- check both redis instances received a
publishevent
Enable the second feature flag:
- run
rails console, and inside it:Feature.enable('use_primary_store_as_default_for_action_cable') - restart gdk
- wait until the second redis instance start receiving
subscribecommands - make another comment in one tab and see it appears on the second one
- check both redis instances received a
publishevent
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.