dev.gitlab.org sidekiq queues growing

Sidekiq queues on dev.gitlab.org have been growing all day. What I first checked them this morning there were 4000 waiting, but now at the end of my day its up over 20000. And the Sidekiq latency for the pipeline_build_hooks queue is now over 10hrs.

The queues are being processed, and I'm not seeing many failures/retries. But we are seeing more events being added than processed.

I haven't been able to figure out what is going on. I'm not seeing a large amount of new builds coming in from postreceive workers, so I have no idea where the events are coming from. They are mostly build_hook, runner cache clearing, and pipeline stage update jobs.

cc\ @marin