## Problem to Solve Deleting a Terraform state file in the GitLab managed state store is a permanent, unrecoverable action. If a state file is deleted due to human error, there is no native mechanism to restore it. For production infrastructure managed via Terraform, an accidental deletion can result in significant operational impact — loss of state means loss of the source of truth for deployed resources, which can block changes, cause drift, or require expensive manual state reconstruction. This gap affects both self-managed and GitLab.com SaaS customers. Self-managed instances have a partial workaround via backup restore (terraform_state.tar.gz) combined with knowledge of the Lockbox encryption scheme, but this requires database access and is not a user-facing recovery path. For SaaS customers, no recovery option exists today. ### Original problem statement Currently, deleting a terraform state in the gitlab state store is a permanent action, as it cannot be recovered afterwards. That is potentially dangerous and could lead to some major impact in production systems, if it's deleted due to human error. It would be great if there would be a "delayed deletion" feature for those as well, similarly to the delayed deletion of projects. ## Proposal Add a delayed deletion mechanism for Terraform state files, mirroring GitLab's existing delayed project deletion. When a state file is deleted, it enters a grace period during which it can be restored before permanent removal. ## Current behaviour https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88496 added a `deleted_at` timestamp so that state files could be removed from object storage before the database records are deleted, because `terraform_state_versions` rows are removed by a cascading foreign key that does not fire ActiveRecord callbacks. `TriggerDestroyService` sets `deleted_at` and immediately enqueues `DestroyWorker`, which removes the files and destroys the record. So `deleted_at` is a work-queue marker, not a recovery window. There is no delay between the two steps. ## Implementation approach ### 1. Do not free the state name during the grace period (Keep the original state name) `terraform_states` has a unique index on `(project_id, name)`, and `RemoteStateHandler#create_or_find!` raises `StateDeletedError` (HTTP 422) when it finds a row with `deleted_at` set. Keep this behaviour: no rename, no index change. A state name is a Terraform backend address, not a UI label. If the name were freed, the next `terraform apply` against the same backend would create a fresh empty row and plan to recreate every resource; on a scheduled pipeline with auto-apply, that duplicates infrastructure before the user can react. There is also no way to restore the old state after the name has been reclaimed. The 422 is the correct failure mode. Match the project deletion flow for permanent removal: `Delete permanently` is available only on a state that is already marked for deletion, and there is no one-shot bypass at delete time. A user who wants to reuse the name immediately performs `Remove state`, then `Delete permanently` from the row in its deleted state, symmetric with `Restore` which is also only available in that state. Skip the grace period entirely for states that have no versions pushed. A `terraform init` may create a state that is never actually used; there is nothing to lose by removing it immediately, and blocking that flow would penalize experimentation during initial setup. Renaming was rejected: it defeats the safety property, and GitLab has no state rename operation, so a restored state would be stuck at a name that no longer matches the user's backend. ### 2. Fixed seven day grace period, no application setting A constant, matching delayed project deletion. This avoids two migrations, a JSONB schema update, an admin form, and the Cells classification, none of which add user value. A setting can be added to an existing JSONB column later if it is asked for. ### 3. Destroy from a daily cron worker Drop the immediate enqueue in `TriggerDestroyService`, add a cron worker that enqueues `DestroyWorker` for states past the grace period, and keep a time check in `DestroyService` as a second line of defence. The cron also covers the retry case that the immediate enqueue was there for. This makes the restorable window and the destruction window mutually exclusive, so a partially destroyed state can never be restored. `DestroyService` should re-check `deleted_at` after loading the record to close the boundary race. ### 4. GraphQL mutation only Add `Mutations::Terraform::State::Restore` alongside the existing `Delete`, `Lock` and `Unlock`. The UI already uses GraphQL, so nothing else is needed to ship. Do not add this to `lib/api/terraform/state.rb`. That namespace is the Terraform HTTP backend protocol, every route is nested under `:id/terraform/state/:name`, and a state named `restore` would collide. Restore needs the same permission as deletion, plus a granular token definition under `config/authz/permissions/terraform_state/`. Protection rules should not be checked, since restoring is not destructive. ### 5. Extend the existing states table `states_table.vue` already branches on `item.deletedAt` to show a "Deleting" badge, and `StateType` already exposes `deleted_at`. Change the badge to show the scheduled deletion date and add a restore action to the row. No new page or field. This is needed either way: once states linger for seven days, the current badge is misleading. ### 6. Feature flag A `gitlab_com_derisk` flag with a project actor, disabled by default. With it off, deletion behaves exactly as it does today. ## Before enabling on GitLab.com Purge existing soft deleted records first. Some may have had part of their version files removed by a failing worker, and would otherwise appear as restorable. ## Suggested breakdown 1. Feature flag; grace period check in `DestroyService`; model scope `ready_for_destruction`. With the flag off, deletion behaves exactly as it does today. 1. Daily cron worker to enqueue states past the grace period; drop the immediate enqueue in `TriggerDestroyService`, and short-circuit for states with no versions (deleted inline, no grace period). Required before the feature flag can be enabled. 1. Restore service, `TerraformStateRestore` mutation, granular permission, and audit event. 1. `Delete permanently` (available only from an already-deleted state, mirroring project deletion): `TerraformStateDeletePermanently` mutation, permission, `DestroyWorker` path that bypasses the grace period, UI action next to Restore, confirmation modal, and audit event. 1. UI: badge with the scheduled deletion date, restore action, permanent-delete action, and updated wording on the existing `Remove state` modal (currently says "cannot be undone", which is no longer true). 1. User documentation. ## Relationship to protected Terraform states Protection rules (https://gitlab.com/gitlab-org/gitlab/-/issues/227108, https://gitlab.com/groups/gitlab-org/-/epics/15118) shipped behind the `protected_terraform_states` flag in 18.11, but the flag has never been enabled, rollout issue https://gitlab.com/gitlab-org/gitlab/-/issues/593962 has no completed steps as of 19.4, and there is no UI or user documentation. They are also only enforced in the REST API: the `TerraformStateDelete` mutation behind the UI's "Remove state" button does not check them. The two solve different problems anyway. Protection stops someone who should not be making the change; delayed deletion recovers from a change made by someone who was allowed to. Neither depends on the other. Version level rollback (https://gitlab.com/gitlab-org/gitlab/-/issues/351295) remains out of scope. ## Community Contribution Note All six steps are suitable for community contribution and can be split across separate MRs. Step 4 (`Delete permanently`) is a moderate task on its own and can be picked up independently once step 1 is merged.