Add Prometheus histogram for PEP worker duration
What does this MR do and why?
Adds a Prometheus histogram that tracks how long it takes us to schedule due pipeline execution project schedules. Our ScheduleWorker executes periodically, and there should be no parallel instances of it executing. We acquire a lock with a 5 minute TTL to avoid iterating the table in parallel. The histogram in this MR has an upper bucket of 300 (5 minutes) and to alert on excessive worker durations, we consider the difference to the implicit Inf+ bucket: gitlab-com/runbooks!8757 (merged)
References
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #512920 (closed)
Edited by Dominic Bauer