PeriodicExclusiveDaemon Application Infrastructure

This is a placeholder issue.

The are several places in the application that periodically wake, read a value, and write a metric observation.

This is used in our elasticsearch infrastructure to monitor queue lengths as one example.

Sometimes the read operations can be expensive, especially some database calls.

Having a mechanism for exclusively locking around these monitors would be helpful and would allow application developers to develop more efficient code.

The API could extend Gitlab::Daemon in the following way:

class Gitlab::ElasticSearchSampler < PeriodicExclusiveDaemon

  def initialize()
    super(interval: 15.seconds)
  end

  # This method will get called once every ~15 seconds (not guaranteed to be exact)
  # across the whole cluster 
  def sample() 
     # Execute a database query, update a metric
  end
end

Must Haves

1. Should be resilient enough to handle restarts across the cluster - exclusivity is "earned" on each invocation, rather than at startup
2. Can rely on Redis Locking infrastructure via redis-persist

Nice to haves

1. "Primary" stickiness. Once a process becomes primary for a PeriodicExclusiveDaemon instance, it has priority on the next invocation. If the primary doesn't obtain the lock within X seconds of the expected time, secondaries can contend to become the new primary.
   • Thinking about this a little more: from a Prometheus point of view, having this would be quite important