2019-07-04 High CPU load on persistent Redis cluster

This has been going on for a while now:

perf top showed this:

A quick traffic analysis showed this:

3519 command: sadd resque:gitlab:queues
3301 command: eval local key, uuid = KEYS[1], ARGV[1]
 819 command: sscan resque:gitlab:processes
 808 command: publish workhorse:notifications
 403 command: srem MIRROR_PULL_CAPACITY
 402 command: lpush resque:gitlab:queue:elastic_commit_indexer

I think sadd resque:gitlab:queues is just a Sidekiq client creating the queue each time it pushes a new job onto the queue?

The second one looks like the LUA script with an ExclusiveLease.

/cc: @andrewn, @vsizov