Decouple the post-deploy migration execution from patch releases
## Context
GitLab [patches](https://handbook.gitlab.com/handbook/engineering/releases/#patch-releases-overview) are SLO-driven releases that deliver bug and security fixes to customers based on [Maintenance Policy](https://docs.gitlab.com/policy/maintenance/). In the run-up to the patch release due date, security fixes are merged and deployed to GitLab.com.
In light of a production incident caused by a security fix, different remediation options are evaluated to restore the customer's operations and GitLab.com's availability.
## Problem: Production rollbacks are not available during the patch release window
Currently, security fixes can include post-deployment migrations. These migrations represent breaking points: when executed in a production environment, they establish a point of no return in the application lifecycle. This represents a high risk for GitLab.com, during production incidents rollbacks to a previous state are unavailable during patch release windows.
Without rollbacks, the mitigation option available is to roll forward: prepare and deploy a new fix. This option takes at least 6 hours and is subject to external factors (pipelines velocity, broken master, flaky failures, etc) [^1].
Waiting hours to restore GitLab.com availability is not acceptable, it is a suboptimal experience for our customers.
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19280 is the last production incident with no rollback available due to an unrelated post-deployment migration execution.
## Proposal: Decouple the post-deploy migration execution from patch releases
By nature, security fixes should be [atomic](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/engineer.md?plain=1#L204), idempotent and they should not represent a breaking point on the application lifecycle. From GitLab historical data, post-deployment migrations are a rare pattern in security fixes:
* [\~1,274 security fixes](https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/?sort=merged_at_desc&state=merged&target_branches%5B%5D=master&first_page_size=20) have been released via GitLab patches (data through 2026-06-03)
* \~1.3% of security fixes have included post-deployment migrations (16 direct security-fix MRs out of \~1,274; the table below also lists 9 mirror-syncs of canonical changes for transparency)
* Breakdown of the 16 by [type](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21612#note_3340915317):
- :green_circle: **Type 1** — housekeeping (block, no friction): **6**
- :yellow_circle: **Type 2** — cleanup of existing data (block + PSIRT sign-off on high severity): **9** (3 are severity::1 or severity::2 — where Felix's gap-window concern applies most directly)
- :red_circle: **Type 3** — migration is the fix itself (escalation path via label-bypass): **1** (`https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3962`)
* Post-deployment migrations on these fixes include DML and DDL operations.
Security vulnerabilities should be addressed at an application level adding and removing indexes or foreign keys likely doesn't have an impact on the vulnerability remediation. The same goes for background migrations adjusting data should be done in a clean-up phase after the fix with the vulnerability has been addressed.
Excluding post-deployment migrations from security fixes will guarantee rollback windows during patch releases, increasing the GitLab.com reliability.
<details>
<summary>Data</summary>
_Data through 2026-06-03_
| Security merge request | Severity | Type | Database operation |
|------------------------|----------|------|--------------------|
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/6176 | | — | N/A - Mirror sync |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/6076 | | — | N/A - Mirror sync |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/5597 | | — | N/A - Mirror sync |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/5368 | | — | N/A - Mirror sync |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/5285 | | — | N/A - Mirror sync |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/5089 | ~"severity::3" | :yellow_circle: Type 2 | DDL (column default) + DML — change `wiki_page_max_content_bytes` default to 5 MB and trim existing instances |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/4754 | ~"severity::2" | :yellow_circle: Type 2 | Background migration — downgrade organization owners |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/4457 | | — | N/A - Mirror sync |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/4080 | ~"severity::3" | :green_circle: Type 1 | Adds indexes and foreign keys |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/4023 | ~"severity::3" | :green_circle: Type 1 | Adding database constraints |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3962 | ~"severity::2" | :red_circle: Type 3 | Updating data |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3041 | ~"severity::3" | :yellow_circle: Type 2 | Background migration — nullify leaked passwords from `ProjectImportState#last_error` |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/2948 | | — | Mirror sync |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/2946 | | — | Mirror sync |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/2573 | ~"severity::3" | :green_circle: Type 1 | Adding and removing indexes |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/2143 | | — | Mirror sync |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1749 | ~"severity::3" | :yellow_circle: Type 2 | Clean up data |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1652 | ~"severity::3" | :yellow_circle: Type 2 | Background migration — correct external-status on existing project bots |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1208 | ~"severity::2" (production incident) | :green_circle: Type 1 | Fixes existing post-deploy migrations |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1207 | ~"severity::2" (production incident) | :green_circle: Type 1 | Revert merge request |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1095 | ~"severity::2" | :green_circle: Type 1 | Modify existing post-deploy migration — caused a production incident |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1066 | ~"severity::3" | :yellow_circle: Type 2 | Background migration — remove existing todos pointing to inaccessible confidential epics |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/417 | ~"severity::1" | :yellow_circle: Type 2 | Background migration — refresh ProjectAuthorization on group deletion (third run) |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/272 | ~"severity::3" | :yellow_circle: Type 2 | Background migration — recalculate ProjectAuthorizations (second run) |
| https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/202 | ~"severity::1" | :yellow_circle: Type 2 | Background migration — recalculate ProjectAuthorizations for shared groups |
<details>
<summary>Script</summary>
```ruby
module ReleaseTools
module Security
class PostDeploymentMigrationsMergeRequests
include ::SemanticLogger::Loggable
def initialize
@client = ReleaseTools::GitlabClient
@result = []
end
def execute
merge_requests.each do |merge_request|
diffs = merge_request_diffs(merge_request.iid)
next if diffs.empty?
if includes_post_deploy_migrations?(diffs)
logger.info('Security merge request includes post-deploy migrations', web_url: merge_request.web_url)
@result << merge_request
else
logger.info('Security merge request does not include post-deploy migrations', web_url: merge_request.web_url)
end
end
@result
end
private
attr_reader :client
def project
ReleaseTools::Project::GitlabEe
end
def merge_requests
Retriable.with_context(:api) do
client.merge_requests(project.security_path, { state: 'merged', target_branch: 'master', page: 10, per_page: 100 })
end
end
def merge_request_diffs(merge_request_iid)
logger.info('Fetching diffs for merge_request', iid: merge_request_iid)
Retriable.with_context(:api) do
client.merge_request_diffs(project.security_path, merge_request_iid)
end
end
def includes_post_deploy_migrations?(diffs)
diffs.map(&:new_path).grep(/post_migrate/).any?
end
end
end
end
```
</details>
</details>
## Implementation approach
The implementation went through two designs. Full design discussion + live-test evidence on https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22084.
**Plan A (superseded):** CODEOWNERS sections on the security mirror requiring Delivery + PSIRT approval on `db/post_migrate/*`. Not viable because:
- `code_owner_approval_required` is `false` on the security mirror's protected branches as a deliberate project-wide pattern.
- Diverging the security mirror's CODEOWNERS from canonical creates a mirroring conflict.
**Plan B (adopted):** Extend the Danger rule itself to verify the bypass label's applier.
- Default-block fires on any security MR touching `db/post_migrate/*`.
- If the bypass label `patch-release::pdm-approved` is applied, the rule fetches `resource_label_events` via `gitlab.api.merge_request_label_events`, identifies the applier, and checks membership against three authorized groups: `@gitlab-org/release/managers`, `@gitlab-org/delivery`, `@gitlab-com/gl-security/product-security/psirt-group`.
- Unauthorized applier → label is removed via `gitlab.api.update_merge_request(remove_labels:)` and the check fails with a link to the runbook.
- Authorized applier → the rule warns and lets the MR through, with an audit note posted as a separate discussion thread via `gitlab.api.create_merge_request_discussion` so the approval event is a permanent, attributable record (independent of Danger's edit-in-place summary comment).
Plan B composes with the existing project-level PSIRT approval rule on the security mirror — that rule continues to require 1 PSIRT approval on every security MR; the Danger rule layers PDM-specific gating on top.
Validated end-to-end on https://gitlab.com/anganga/pdm-danger-test/-/merge_requests/2. Implementation tracker: https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21612.
## Exit criteria
- [x] Delivery and PSIRT agreed on not having post-deploy migrations on security fixes
- [x] Post-deploy migrations are not allowed on security fixes
## Plan
### Meta
| Status | Issue |
|--------|-------|
| :white_check_mark: | Share proposal with PSIRT to gather their thoughts and approval |
### Implementation details
| Status | Issue |
|--------|-------|
| :white_check_mark: | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/22084+ |
| :white_check_mark: | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21615+ |
| :white_check_mark: | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21612+ |
| | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21613+ |
### Rollout
| Status | Issue |
|--------|-------|
| :white_check_mark: | Ensure Release & Deploy team members are aware of this |
| | Broadcast this change to Engineering |
<details>
<summary>Drawer of thoughts</summary>
in the event of a production incident.during the patch release preparation, post-deploy migrations are executed on GitLab production environments.In light of a production incident caused by security fixes, security fixes can't be
To comply with [GitLab SLAs](https://handbook.gitlab.com/handbook/security/product-security/vulnerability-management/sla/) vulnerabilities, [patches are released](https://handbook.gitlab.com/handbook/engineering/releases/#patch-release-cadence) twice a month on Wednesdays surrounding the release week. To guarantee patch availability on the target date, release managers start the patch release preparation the Monday before, merging and deploying security fixes to GitLab.com.
Because security fixes can include post-deploy migration, the post-deploy migration execution is tied to the patch release process, during the patch release preparation, post-deploy migrations are executed on GitLab production environments.Waiting 6 hours to restore the customer's availability is unacceptable.To fulfill the target date, release managers start a preparation the previous Monday.a scheduled cadence to self-managed and Dedicated customers. Post-deployment migrations represent application breaking points, when added into the development cycle, a post-deployment pipeline execution is required, this prevents our ability to rollback a production deployment.
Due to their private nature, [security fixes are discouraged from reverting](https://handbook.gitlab.com/handbook/engineering/releases/patch-releases/#how-can-i-revert-a-security-merge-request).
</details>
## Admin
```
/label ~"Delivery::P4" ~"post-deploy migrations" ~"group::release-and-deploy" ~"release tooling"
/epic &1755
```
## Status Notes
<!-- STATUS NOTE START -->
## Status 2026-07-30
:clock1: **total hours spent this week by all contributors**: 16h
:tada: **achievements**:
- Backported the Danger rule to all maintained versions (19.2, 19.1, 19.0). After confirmation that the rule should ship on stable branches for maintenance consistency (even though it is a no-op there), all three backport MRs were merged. https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21612 is now closed.
- The runbook merged, documenting the policy, the alternatives, and the exception process. https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21615 is now closed.
- Started the last child issue, https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21613: opened https://gitlab.com/gitlab-org/release-tools/-/merge_requests/5168 to remove the now-redundant post-deploy migration step from the security patch template, plus a follow-up (https://gitlab.com/gitlab-org/release/docs/-/merge_requests/1093) adding an execution reminder to the runbook's exception process, so an approved-exception migration still gets run once the standing template step is gone.
:issue-blocked: **blockers**:
- None. Awaiting review and merge on the last two open MRs above.
:arrow_forward: **next**:
- Merge the two MRs and close https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21613.
- Broadcast the change to Engineering (rollout), then close the epic.
_Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1755#note_3620716524_
<!-- STATUS NOTE END -->
[^1]: Hot patch is another remediation option but pending on the service impacted it may not be available.
epic