Introduce separate sidekiq shard for mailers queue
Two years ago we rolled out single queue per sidekiq shard: &469 (closed). As part of that, we made every shard BRPOP only from a single Redis queue, with one exception: mailers
.
The catchall
shard BRPOPs on both default
and mailers
: &447 (closed).
We can see that this is the only shard with more than one queue, and it happens to be the most active one:
➜ tmp cat trace.json | grep brpop | jq -c '.keys|sort' | sort | uniq -c | sort -rn
49803 ["queue:default","queue:mailers"]
12837 ["queue:urgent_other"]
8784 ["queue:urgent_cpu_bound"]
4339 ["queue:low_urgency_cpu_bound"]
1294 ["queue:elasticsearch"]
1037 ["queue:urgent_authorized_projects"]
883 ["queue:quarantine"]
291 ["queue:memory_bound"]
48 ["queue:gitaly_throttled"]
30 ["queue:database_throttled"]
Since we determined that reducing the number of arguments to BRPOP greatly reduces CPU cost, it may be worth creating a dedicated shard for mailers
so that each of them has a single-queue BRPOP.
We may see diminishing returns at this point, as going from 2 queues to 1 may not make as big of a difference. It'd be good to find out though.
I was not able to find the reasoning behind not splitting this when we did the original 1 queue per shard migration. If anyone knows of a technical reason, I'm all ears.