Skip to content

Introduce Pubsub redis instance

Sylvester Chin requested to merge sc1-migrate-pubsub into master

What does this MR do and why?

This MR introduces Gitlab::Redis::Pubsub to allow us to migrate actioncable's pubsub operation across Redises.

See gitlab-com/gl-infra/scalability#2489 (closed)

Pubsub reads from config/redis.yml's pubsub key while actioncable loads its Redis configuration from config/cable.yml.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Set up gdk start.

  1. Configure your config/redis.yml file and start a redis-server. We start a separate server since pubsub occurs across a server's db.
---
development:
  pubsub: "redis://localhost:6379"

  1. Ensure both feature flags are disabled

  2. Open 2 pages of http://gdk.test:3000/gnuwget/Wget2/-/issues/13. Check pubsub connections via:

➜  gitlab git:(sc1-migrate-pubsub) ✗ gdk redis-cli pubsub channels
 1) "development_:graphql-subscription:f1956553-afc3-443d-86ab-d5eed1918cfe"
 2) "development_:action_cable/Z2lkOi8vZ2l0bGFiL1VzZXIvMQ"
 3) "development_:graphql-event::issuableEpicUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
 4) "development_:graphql-event::issuableDatesUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
 5) "development_:graphql-subscription:5a214d64-8273-4ae3-9c85-41d811836fc2"
 6) "development_:graphql-event::issuableWeightUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
 7) "development_:graphql-event::issuableAssigneesUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
 8) "development_:graphql-subscription:0c72f3f8-1c7c-4224-b2d0-8945dfc9697a"
 9) "development_:graphql-subscription:d5bb66e4-8422-40dc-8bab-a376474a6fc8"
10) "development_:graphql-subscription:c787895d-dc25-423b-a037-0443bb2f148a"
11) "_action_cable_internal"
12) "development_:graphql-event::issuableLabelsUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
  1. Enable dual-write via the following command. Wait ~1min for in-mem cache to invalidate.
Feature.enable(:use_primary_and_secondary_stores_for_action_cable)
  1. On one of the 2 issues page, perform any of the following actions
  • add/remove labels
  • assign/unassign

The other issues page will sync.

Confirm double-write by subscribing to the channels on the primary store. You should see messages related to your action, in this case, it is label updates.

redis-cli  subscribe development_:graphql-subscription:50aeb92f-61ee-40d7-80c6-5819c01fe9c5 development_:graphql-subscription:f1956553-afc3-443d-86ab-d5eed1918cfe development_:action_cable/Z2lkOi8vZ2l0bGFiL1VzZXIvMQ development_:graphql-event::issuableEpicUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw development_:graphql-subscription:d25cf163-0db2-47c1-afd0-9f88f283f563 development_:graphql-event::issuableDatesUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw development_:graphql-subscription:5a214d64-8273-4ae3-9c85-41d811836fc2 development_:graphql-subscription:fa2d4158-fdb7-495f-a214-8c22f67f0dfe development_:graphql-subscription:97ecdf75-01e3-4225-83ee-dbca91e719f9 development_:graphql-event::issuableWeightUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw development_:graphql-event::issuableAssigneesUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw development_:graphql-subscription:0c72f3f8-1c7c-4224-b2d0-8945dfc9697a development_:graphql-subscription:d5bb66e4-8422-40dc-8bab-a376474a6fc8 development_:graphql-subscription:c787895d-dc25-423b-a037-0443bb2f148a development_:graphql-subscription:623f4797-91a1-4745-b97f-c5848da078f7 _action_cable_internal development_:graphql-event::issuableLabelsUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw

...

1) "message"
2) "development_:graphql-event::issuableLabelsUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
3) "\"{\\\"wal_locations\\\":{},\\\"gql_payload\\\":{\\\"__gid__\\\":\\\"Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw\\\"}}\""
1) "message"
2) "development_:graphql-subscription:f1956553-afc3-443d-86ab-d5eed1918cfe"
3) "{\"result\":{\"data\":{\"issuableLabelsUpdated\":{\"id\":\"gid://gitlab/Issue/40\",\"labels\":{\"nodes\":[{\"id\":\"gid://gitlab/ProjectLabel/89\",\"title\":\"Bronco\",\"description\":null,\"color\":\"#9664c3\",\"textColor\":\"#FFFFFF\",\"__typename\":\"Label\"},{\"id\":\"gid://gitlab/ProjectLabel/87\",\"title\":\"Escalade\",\"description\":null,\"color\":\"#3ba8e6\",\"textColor\":\"#FFFFFF\",\"__typename\":\"Label\"}],\"__typename\":\"LabelConnection\"},\"__typename\":\"Issue\"}}},\"more\":true}"
1) "message"
2) "development_:graphql-subscription:97ecdf75-01e3-4225-83ee-dbca91e719f9"
3) "{\"result\":{\"data\":{\"issuableLabelsUpdated\":{\"id\":\"gid://gitlab/Issue/40\",\"labels\":{\"nodes\":[{\"id\":\"gid://gitlab/ProjectLabel/89\",\"title\":\"Bronco\",\"description\":null,\"color\":\"#9664c3\",\"textColor\":\"#FFFFFF\",\"__typename\":\"Label\"},{\"id\":\"gid://gitlab/ProjectLabel/87\",\"title\":\"Escalade\",\"description\":null,\"color\":\"#3ba8e6\",\"textColor\":\"#FFFFFF\",\"__typename\":\"Label\"}],\"__typename\":\"LabelConnection\"},\"__typename\":\"Issue\"}}},\"more\":true}"

Read cut-over

  1. Enable primary store as default and restart rails

In Rails console:

Feature.enable(:use_primary_store_as_default_for_action_cable)

In terminal:

gdk restart rails
  1. Check that subscribers are using the new store
➜  gitlab git:(sc1-migrate-pubsub) ✗ gdk redis-cli pubsub channels
(empty array)
➜  gitlab git:(sc1-migrate-pubsub) ✗ redis-cli pubsub channels
 1) "development_:graphql-event::issuableWeightUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
 2) "development_:graphql-subscription:2bd24945-94a7-4481-b5f1-edc0cbf47961"
 3) "development_:graphql-event::issuableDatesUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
 4) "development_:graphql-subscription:864bf905-d1f8-49c3-a255-bfab978f2ba2"
 5) "development_:graphql-event::issuableLabelsUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
 6) "_action_cable_internal"
 7) "development_:graphql-event::issuableEpicUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
 8) "development_:graphql-subscription:d74d86f4-5d10-40cf-9c60-20025cf7b77e"
 9) "development_:graphql-subscription:8c5c2635-9fb2-49c5-82c3-b80600299891"
10) "development_:graphql-event::issuableAssigneesUpdated:issuableId:Z2lkOi8vZ2l0bGFiL0lzc3VlLzQw"
11) "development_:action_cable/Z2lkOi8vZ2l0bGFiL1VzZXIvMQ"
12) "development_:graphql-subscription:2f4ec679-12b0-4f3a-bc25-726f023ea153"
13) "development_:graphql-subscription:aa5fe520-53c1-4131-9e4c-c75fa965e721"
14) "development_:graphql-subscription:5ff2b4af-1ee1-4918-9ca4-0a4d17334a33"
15) "development_:graphql-subscription:20bbcb63-d64c-49e6-aeb4-603e035972ee"
16) "development_:graphql-subscription:2d1656b5-1fd1-4e5c-8eb9-31d4b99da854"
17) "development_:graphql-subscription:77e9dbe3-ab06-4d15-8435-91c39ade9714"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sylvester Chin

Merge request reports