[backend] Add PipelineExecutionPolicies::ScheduleWorker execution duration metric and alerting

In !175246 (merged) we are introducing the periodic background worker Security::PipelineExecutionPolicies::ScheduleWorker. The worker gets enqueued every minute and acquires a lock to prevent multiple jobs racing in parallel to perform due project schedules. The lock currently has a TTL of 5 minutes, which means that a background job's runtime must not exceed 5 minutes to prevent the race condition, or else a duplicate job may get enqueued

We need to:

  • add metrics/a histogram for the worker execution duration
  • add an alerting rule that fires when the worker exceeds our TTL
Edited by Dominic Bauer