Process pending Incident Escalations

What does this MR do and why?

This MR adds processing of incident escalations to on-call users, as per the Issue #330281 (closed).

It does not cover the setting up/creation of these escalations.

Included in this MR scope:

• Checks the Pending Escalation to determine if it should be escalated, in IncidentManagement::PendingEscalations::IssueCheckWorker
• Modified IncidentManagement::PendingEscalations::ProcessService to handle PendingEscalations::Issue records
• Add Email template for when incident is escalated

This feature is behind a new feature flag, incident_escalations.

Excluded from this MR (to be in subsequent MRs)

• Create Incident escalation on the creation of an incident
  • Creates an IncidentManagement::IssuableEscalationStatus on the Incident (done in: !75176 (merged))
  • Create an IncidentManagement::PendingEscalations::Issue record
• Removing the escalation when an Incident is converted to an Issue
• Creating an escalation when an Issue is converted to an Incident
• Handling logic when Issue is moved projects

Screenshots or screen recordings

Emails

Plain	HTML

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

1. Enable the feature flag

   Feature.enable(:oncall_schedules)
   Feature.enable(:escalation_policies)
   Feature.enable(:incident_escalations)

2. Create an Escalation Policy and rule via Monitor -> Escalation Policies
3. Create an Incident via Issues -> Issue Type 'Incident'.
4. Create an Escalation manually using the following commands:

# Fetch data
incident = Issue.incident.last
policy = incident.project.incident_management_escalation_policies.first

# Create pending escalations
rules = policy.active_rules
rules.map do |rule|
  IncidentManagement::PendingEscalations::Issue.create!(
    target: incident,
    rule: rule,
    process_at: rule.elapsed_time_seconds.seconds.after(Time.current)
  )
end

# Create escalation status
IncidentManagement::IssuableEscalationStatus.create!(issue: incident, policy: policy, escalations_started_at: Time.current)


# Queue job to process escalations
IncidentManagement::PendingEscalations::ScheduleCheckCronWorker.new.perform

1. Once created, your incident should escalate to the user specific in your Escalation Policy rule via email. (See screenshots above)
2. A system note should also be added to the Incident Issue

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #330281 (closed)