Add explicit guidance for ICs under the 24 hour SLO for flaky test remediation

GitLab uses custom tooling to automatically identify and report the most impactful flaky test files that block CI/CD pipelines. The [ci-alerts automation](https://gitlab.com/gitlab-org/quality/analytics/ci-alerts) creates issues for test files causing repeated pipeline failures, which are then triaged and assigned to Engineering Managers for resolution.

**View all top flaky test file issues:** [automation:top-flaky-test-file label](https://gitlab.com/gitlab-org/quality/test-failure-issues/-/issues/?label_name%5B%5D=automation%3Atop-flaky-test-file)

**View all top flaky test file issues:** [automation:top-flaky-test-file label](https://gitlab.com/gitlab-org/quality/test-failure-issues/-/work_items?sort=created_date&state=opened&label_name%5B%5D=automation%3Atop-flaky-test-file&not%5Blabel_name%5D%5B%5D=quarantine&not%5Blabel_name%5D%5B%5D=flaky-test%3A%3Afalse-positive&first_page_size=20)

**Top flaky test file issues open >24 hours:** [GLQL Tracking Issue](https://gitlab.com/gitlab-org/quality/test-failure-issues/-/work_items/43806)

**Flaky test file issues that remain open for longer than 24 hours will be escalated as production incidents.**

### How It Works

For guidance on quarantining tests, see the [Quarantine Process (Handbook)](../quarantine-process/) and [Quarantining Tests (Developer Docs)](https://docs.gitlab.com/development/testing_guide/quarantining_tests/).

### Urgency Tiers and Response Timelines

### For Assigned ICs

Engineering managers will assign flaky test issues to an IC with immediate capacity. As an IC, if you have capacity and see one for your team unassigned, pick it up immediately for development. When you are assigned to a flaky test issue, you are expected to drop what you're doing and fix the test today.

In service of resolving these issues immediately, please also drop what you're doing to review any of these fixes for someone else who's assigned to the failing test. **This is synchronous work;** ping people directly, have the code reviewed immediately, get your approvals and merge your changes.

Lean hard on the agents to get these fixes written and merged. When unblocking all of the rest of development on the gitlab project, speed is critical!

#### Workflow

1. A flaky test issue is opened in status **New** and assigned to the attributable EM.

2. An IC picks up the issue, or is assigned by an EM. The issue and any sub-tasks should worked on immediately, set to the current milestone, and moved to **In Development**.

3. The review phase for these MRs should be very short. Assigned ICs should be working directly and synchronously with at least one other developer to merge their fixes.

4. If an engineer's day ends and there are still changes to be made, the assigned engineer is responsible for handing off the work to someone in another time zone to be assigned and continue working. These issues and MRs are not to be left idle overnight.

5. When enough fixes have been made that the test should no longer be blocking other developers, the flaky test file issue should be moved to **Verification**.

6. After **3 days** without a failure, the DevEx flaky test automation will close the issue.

#### Urgency Tiers and Response Timelines

Flaky tests are categorized by urgency based on their impact on pipeline stability:

Teams have 24 hours from the flaky test file being identified to ship their fixes, after which the issue will be escalated to a Severity 3 production incident and require the post-incident review process.

- 🔴 **Critical**: 48 hours - Tests blocking critical workflows, deployment pipelines or affecting multiple teams

- 🟠 **High**: 1 week - Tests with significant pipeline impact

- 🟡 **Medium**: 2 weeks - Tests with moderate impact

+After X (TBD) hours at a Severity 3, the incident will escalate to a Severity 2.

These timelines guide when a test should be quarantined if it cannot be fixed. For quarantine procedures and technical implementation, see [Quarantine Process (Handbook)](../quarantine-process/) and [Quarantining Tests (Developer Docs)](https://docs.gitlab.com/development/testing_guide/quarantining_tests/).