Create a curated list of interactions with redis-sidekiq
Which components (clients, schedulers, fetchers, processors, middlewares, APIs, etc.) are interacting with redis-sidekiq? And how do they do that? Apart from Sidekiq's internal components, we implemented many components that interact with Redis sidekiq. Some examples are SidekiqStatus middleware, Deduplication middleware, etc. Besides, we do re-implement a part of Sidekiq, such as Semi-reliable fetcher. In this fetcher, we introduced a new InteruptedSet concept. It's really hard to split something we don't know in details how it interacts with others. Therefore, this prerequisite is to list out all the components we are using on production, as detailed as possible.
Scanning through all possible locations is inevitable. However, we may miss some components, especially the ones that run infrequently or conditionally. One good solution for this is to analyze the live traffic of redis-sidekiq. In detail, we need to run tcpdump on redis-sidekiq server, feed the resulted pcap file to tcpflow command, and then run the redis trace analyzer script. The final files contain the commands, normalized key patterns, and even request and response sizes of the traffic. They can be used to double-checked the manual analytics we conduct. The samples should be taken at different periods of the day, especially some special ones like beginning of the day (UTC +0), beginning of hours, etc.
I'll analyze each component, one by one in the comments.
-
✅ Sidekiq client: -
✅ Sidekiq server: -
✅ Cron Jobs -
✅ Middlewares: -
✅ GitLab application components:-
✅ Gitlab::Database::MigrationHelpers -
✅ LimitedCapacity::JobTracker -
✅ Gitlab::BatchPopQueuing -
✅ Gitlab::SidekiqMigrateJobs - used in Sidekiq rake tasks. -
✅ Gitlab::BackgroundMigration::JobCoordinator -
✅ Workers that depend on their own queue sizes or other workers' queue sizes: #995 (closed)
-
-
✅ Administration: -
✅ Miscellaneous components:-
✅ Mailroom, which pushes jobs directly into redis-sidekiq.
-
A complete dependency map from the above components: