## Summary <!-- Give context for what problem this issue is trying to prevent from happening again. Provide a brief assessment of the risk (chance and impact) of the problem that this corrective action fixes, to assist with triage and prioritization. --> https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16042+ and https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24115+ could've benefited from an automated workflow that would halt/rollback deployments based on time correlation. Our bots have auto-started an incident while the deployment was in progress: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16041. Can EOCs initiate deployment rollback to speed recovery time? If not, why not? ## Related Incident(s) <!-- Note the originating incident(s) and link known related incidents/other issues. The relation will happen automatically if you are creating this issue from an incident, if this isn't done already please uncomment the following line: --> Originating issue(s): https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16042+ Also https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24115+ ## Desired Outcome/Acceptance Criteria <!-- How will you know that this issue is complete? If you have any initial thoughts on implementation details (e.g. what to do or not do, gotchas, edge cases etc.), please share them while they are fresh in your mind. --> Multiple ideas: - When an auto-generated incident is opened during an ongoing deploy, the deploy should be halted automatically. - When an auto-generated incident is opened immediately following a deploy, the deploy should be rolled back automatically. - When the root cause of an incident is determined to be a software change in the most recent deploy, an EOC should be able to safely roll it back. ## Associated Services <!-- Apply the appropriate services associated with this corrective action if applicable. reliability~25597333 --> ## Corrective Action Issue Checklist * [x] Link the incident(s) this corrective action arose from * [x] Give context for what problem this corrective action is trying to prevent re-occurring * [x] Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') * [x] Assign a [priority](https://about.gitlab.com/handbook/engineering/infrastructure/team/reliability/issues.html#issue-priority) (this will default to 'Reliability::P4' but should match the severity of the related incident)