Skip to content

Plan rollout for Sidekiq shard router

This issue outlines the approach for the rollout. Finer details (dashboard links and charts) will be in the change issues for gstg and gprd respectively.

Pre-requisites

  1. Observability -- gitlab exporter polling the new Redis instance with sidekiq probes enabled
  2. Application changes -- feature flag definition file for the new shard needs to be merged
  3. Routing rules to be updated
  4. Redis configuration in redis.yml to be updated with the new Redis instance information

3 and 4 will likely require a CR for gprd since it involves merging configuration changes into gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/

Roll-out

  1. Feature flag toggle to 10% and crank up towards 100%. 10->25->50->75->100
  2. General domains to monitor
  • job push and fetches
  • scheduled jobs
  • crons

General job pushes and fetches

.perform_async are performed within a Sidekiq::Client.via block which routes the push to the appropriate Redis pool. This is done on every .perform_async which allows us to control it using feature flags.

On the server-side, the server-metric middleware would add the store label which lets us track the rate of processed jobs for the default queue across the 2 instances.

Scheduled jobs

Scheduled jobs refer to .perform_at/.perform_in. When adding the job into the schedule sorted set, the raw_push is performed within a via block. This means the jobs would be added to the schedule sorted set in the respective sidekiq shard Redis.

We have added a custom scheduler enq class to use the shard router when doing .push (https://gitlab.com/gitlab-org/gitlab/-/blob/d02cd1e539fe70613c61e4964f589fa097afe72b/lib/gitlab/sidekiq_sharding/scheduled_enq.rb#L26).

The schedule set size can be tracked since there is a sidekiq probe for the new Redis instance.

Crons

The new catchall-2 shard will have cron-pollers disabled. So only pre-existing shards will continue to schedule cron-jobs.

However, the cron-pollers will still push jobs through the Sidekiq shard router. This is covered in #2817 (closed).

Edited by Sylvester Chin