Plan rollout for Sidekiq shard router
This issue outlines the approach for the rollout. Finer details (dashboard links and charts) will be in the change issues for gstg and gprd respectively.
Pre-requisites
- Observability -- gitlab exporter polling the new Redis instance with sidekiq probes enabled
- Application changes -- feature flag definition file for the new shard needs to be merged
- Routing rules to be updated
- Redis configuration in
redis.yml
to be updated with the new Redis instance information
3 and 4 will likely require a CR for gprd since it involves merging configuration changes into gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/
Roll-out
- Feature flag toggle to 10% and crank up towards 100%.
10->25->50->75->100
- General domains to monitor
- job push and fetches
- scheduled jobs
- crons
General job pushes and fetches
.perform_async
are performed within a Sidekiq::Client.via
block which routes the push to the appropriate Redis pool. This is done on every .perform_async
which allows us to control it using feature flags.
On the server-side, the server-metric middleware would add the store
label which lets us track the rate of processed jobs for the default
queue across the 2 instances.
Scheduled jobs
Scheduled jobs refer to .perform_at/.perform_in
. When adding the job into the schedule
sorted set, the raw_push
is performed within a via
block. This means the jobs would be added to the schedule
sorted set in the respective sidekiq shard Redis.
We have added a custom scheduler enq class to use the shard router when doing .push
(https://gitlab.com/gitlab-org/gitlab/-/blob/d02cd1e539fe70613c61e4964f589fa097afe72b/lib/gitlab/sidekiq_sharding/scheduled_enq.rb#L26).
The schedule
set size can be tracked since there is a sidekiq probe for the new Redis instance.
Crons
The new catchall-2
shard will have cron-pollers disabled. So only pre-existing shards will continue to schedule cron-jobs.
However, the cron-pollers will still push jobs through the Sidekiq shard router. This is covered in #2817 (closed).