Existing incidents don't have corresponding statuses when :incident_escalations flag is enabled

Context

A bug was raised while testing during the rollout of incident escalations. The feature is still disabled, but it's blocking rollout.

Bug behavior

When the incident_escalations feature flag is not enabled, the escalation status records for incidents are not saved. So when the flag is enabled, existing incidents do not have status records (and attempting to update the values does not save the record). As we've previously backfilled some records, there's just a batch of incidents which have incorrect information & can't be changed by the user.

Why this is happening: !78926 (diffs) refactored Issuable#supports_escalation? to include a feature flag check. But https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/issues/create_service.rb#L90 had previously relied on there being no feature flag check.

Technical considerations

For incident-associated tables, we broadly have 2 choices:

  1. Backfill records and create new records on incident creation or issue-type changes.
  2. Rely on default values & save records when a user interacts with the associated table

grouprespond had previously decided to take approach #1, to make sorting & searching by status simpler, as well as minimizing complexity in syncing escalation-related attributes between alerts and incidents. Allow quick action command /page in issue descr... (#353050 - closed) may call for something closer to #2.

Incident severity follows approach #2, for which the default value on missing records is static ("unknown"). For escalation status, because the status/policy can come from the alert, the default is more complicated.

The escalation status records which have already been backfilled don't actually include information from the alert. However, new incidents created from alerts do have incident information. As a user, I'd expect that if the status values sync between alerts & incidents, the values should be correct from the start.

Plan

Expected behavior
  1. For incidents created after the feature flag is enabled, keep the existing behavior.
  2. For existing incidents with alerts: incident status/policy should match the alert, for consistency with behavior of new incidents and with the alert-incident sync.
  3. For existing incidents without alerts: status/policy will be None. Updates to status will create a new record with the specified status. Setting the policy will create a record with the specified policy, set the status to triggered, and start escalations. This is also then consistent with the behavior of new incidents.
Implementation
  1. Delete existing issuable_escalation_status records (TRUNCATE?)
  2. Backfill issuable_escalation_status records for existing incidents with associated alerts
  3. Update issue creation to create issuable_escalation_status records for new incidents associated with alerts, even if feature flag disabled
  4. Update logic for setting status/policy to save new issuable_escalation_status records if no record exists
Edited by Sarah Yasonik