PeriodicExclusiveDaemon Application Infrastructure
This is a placeholder issue.
The are several places in the application that periodically wake, read a value, and write a metric observation.
This is used in our elasticsearch infrastructure to monitor queue lengths as one example.
Sometimes the read operations can be expensive, especially some database calls.
Having a mechanism for exclusively locking around these monitors would be helpful and would allow application developers to develop more efficient code.
The API could extend Gitlab::Daemon
in the following way:
class Gitlab::ElasticSearchSampler < PeriodicExclusiveDaemon
def initialize()
super(interval: 15.seconds)
end
# This method will get called once every ~15 seconds (not guaranteed to be exact)
# across the whole cluster
def sample()
# Execute a database query, update a metric
end
end
Must Haves
- Should be resilient enough to handle restarts across the cluster - exclusivity is "earned" on each invocation, rather than at startup
- Can rely on Redis Locking infrastructure via
redis-persist
Nice to haves
- "Primary" stickiness. Once a process becomes primary for a
PeriodicExclusiveDaemon
instance, it has priority on the next invocation. If the primary doesn't obtain the lock within X seconds of the expected time, secondaries can contend to become the new primary.- Thinking about this a little more: from a Prometheus point of view, having this would be quite important
Edited by Andrew Newdigate