[backend] Add PipelineExecutionPolicies::ScheduleWorker execution duration metric and alerting
In !175246 (merged) we are introducing the periodic background worker Security::PipelineExecutionPolicies::ScheduleWorker. The worker gets enqueued every minute and acquires a lock to prevent multiple jobs racing in parallel to perform due project schedules. The lock currently has a TTL of 5 minutes, which means that a background job's runtime must not exceed 5 minutes to prevent the race condition, or else a duplicate job may get enqueued
We need to:
- add metrics/a histogram for the worker execution duration
- add an alerting rule that fires when the worker exceeds our TTL
Edited by Dominic Bauer