Investigate if we can optimize CI runner pubsub Redis workload

CI runners use long polling to avoid hammering Rails backends. This is made possible by a stream of pubsub messages sent from Sidekiq (?) to all Workhorse processes. It is a known issue (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12812, #1392 (closed)) that this is one of the largest workloads on redis-persistent.

I want to take another shot at seeing if we can optimize this. I will log and report my findings in this issue.

Status 2022-08-30

It is clear that there is room for improvement: we deliver a firehose of messages to Workhorse processes which drop 99.9% of them. Promoted this issue to epic: &784.