Investigating dangling jobs in redis-sidekiq default queue
GSTG
On gstg, we observe jobs in the queue:default list.
GPRD
After production#17868 (closed), it is observed that there were a few jobs being processed by pods in the gitlab-sidekiq-catchall-v2 deployment even after the feature flag has been fully enabled at 0744h UTC 09 May 2024.
The root-cause will be explained in the comments. But in summary, the behaviour does not indicate a bug in the sharding router.