Handle application functionality that needs to see all Sidekiq Redis instances
In &423, we're broadly assuming that our Rails application needs to only know about a single Sidekiq Redis instance at a time, so we can have shards that have no knowledge of each other. However, there are some cases where we want to do something to all Sidekiq Redis nodes:
Component | Note |
---|---|
Gitlab::SidekiqMigrateJobs | This component iterates through the schedule set and retry set. These sets are sharded. It should visit each instance. |
Workers that depend on their own queue sizes or other workers' queue sizes | The queue size is fetched from Sidekiq APIs. We need federation supports to perform the action correctly. We can either add federation support or refactor them not to depend on the queue size |
API::SidekiqMetrics | As its name already suggests, this component is a wrapper to access Sidekiq data. It uses Queue, ProcessSet, and Stats APIs. |
API::Admin::Sidekiq | This component modifies the jobs upon a queue. It uses Sidekiq APIs exclusively. |
Web UI | This component uses built-in Sidekiq APIs. As it is a rack middleware, we could add a middleware in front of it to force Sidekiq APIs use federation. |
gitlab-exporter | This component is outside of the Rails application. It means it could not reuse anything we implement in GitLab rails codebase. However, if the federation Sidekiq APIs are encapsulated as a gem, we can cut down a lot of effort! |
Rails console | Also mentioned in #935 |
For all of these cases we would need a way to access all Sidekiq Redis instances from a single application instance. However, this has problems of its own:
- It's configuration complexity on the development and administration side. We were hoping this was out of scope for &423.
- Any operations that need to run on all Sidekiq Redis instances will be more complicated. With two Sidekiq Redis-es this is probably OK. What about if we have 10?
- There is also a resiliency question. With completely isolated (application, Sidekiq Redis) pairs that know nothing about each other, one Sidekiq Redis can go down briefly and other jobs will still be processed. But these operations will fail.
Edited by Quang-Minh Nguyen