## Context The current Cloud SQL restore runbook at https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/runway/cloudsql_backup_restore.md#restore describes a restore process that runs `gcloud sql backups restore` directly against the GCP API. This bypasses Runway/Terraform entirely and breaks the single source of truth for TF state, creating drift that is hard to detect and reconcile. Additionally, restoring an instance directly through GCP (outside of Runway/Terraform) means Runway automation cannot reconfigure auxiliary services that depend on the instance — such as Grafana dashboards, Elastic integration, and other observability tooling that Runway sets up as part of the managed lifecycle. Those would be left in an inconsistent or broken state after a direct restore. Reference discussion: https://gitlab.com/gitlab-org/gitlab/-/work_items/565416#note_3626612047 ## Goal Update the Cloud SQL restore runbook so that the restore process goes through Runway MRs rather than direct `gcloud` API calls, with SRE approval as the control gate. Service Owners should still have the permission to trigger a restore, but it must be done through a Runway MR that gets reviewed and approved by SREs, reducing operational risk and keeping TF state consistent. ## Acceptance criteria - [ ] Runbook updated to describe a Runway MR-based restore flow - [x] Direct `gcloud sql backups restore` usage removed or clearly marked as last-resort only - [x] SRE approval step documented as part of the restore process - [ ] Impact on auxiliary services (Grafana, Elastic, etc.) documented and addressed in the runbook ## Default path Rather than a single default, the decision follows two questions in order (assuming 7-day retention and no recovery point older than that): 1. **Is any current data worth keeping?** If yes → recover into a side instance and extract. Never restore prod. 2. **Does the fix need a new instance name?** If yes → pay the full cutover. What falls out of that: - **Most common case** — someone dropped a table, everything else is fine. Restoring prod would throw away every good write since. Default is **side instance + extract** — no cutover, nothing to repoint. - **When nothing current is worth keeping** — restore to a new instance with `clone { point_in_time }` and cut over. - **In-place `restore_backup_context` is always the last resort**, even though it's the only option that avoids a cutover. Cheap isn't the same as safe. ## Cutover ownership Cutover goes through Runway. Owner is the **on-call engineer running the procedure**. Work required depends on the method: | Method | Cutover work | |---|---| | In-place restore | None | | Side instance + extract | None — delete the side instance after | | New instance | Endpoint, DNS and IAM regenerate automatically from the inventory. Manual: OpenBao's `instance_connection_name`, plus service + metrics catalog entries | > **Note:** Grafana is keyed on the **instance name**, and the catalog entries live in the runbooks repo and need `make generate`. A restored instance has no dashboards until that MR lands — pre-stage it rather than write it mid-incident. ## Restore use cases Both lists ordered most to least frequent in real life, with the Terraform field each case needs. ### Disaster recovery | # | Case | Terraform | Cutover | |---|---|---|---| | 1 | Bad migration or deploy; whole DB bad, nothing since worth keeping | `clone { point_in_time }` → new instance + cutover. `restore_backup_context` in place only as a last resort | Full | | 2 | Accidental `DELETE`/`UPDATE`; rest of the DB is fine | `clone { point_in_time }` → side instance → extract | None | | 3 | Accidental `DROP TABLE` | `clone { point_in_time }` → side instance → extract | None | | 4 | Zone failure | **None** — REGIONAL fails over on its own | None | | 5 | `DROP DATABASE` / mass loss | `clone { point_in_time }` → new instance + cutover. `restore_backup_context` in place only as a last resort | Full | | 6 | Storage or index corruption | `clone { point_in_time }` to establish when it started, then new instance + cutover | Full | | 7 | Region outage | `restore_backup_context { backup_run_id, instance_id }` on a new instance in another region — `clone` needs a live source | Full | | 8 | Instance deleted | `restore_backup_context { backup_run_id, instance_id }` on a new instance, **only if** the backups survived (see Gap 1 below) | Full | | 9 | Malicious deletion / compromised creds | **None** — backups are in the same project (Gap 3) | — | Rows 2 and 3 are the bulk of real incidents, and neither needs a cutover. ### Other purposes | # | Case | Terraform | |---|---|---| | 1 | Backup validation | **None** — already automated via `gcloud sql backups restore` in the restore-validation pipeline | | 2 | Forensics: "what did it look like at time T?" | `clone { point_in_time }`, read-only, delete after | | 3 | Pre-upgrade safety copy | On-demand backup, plus `clone` (no `point_in_time`) for a warm rollback target | | 4 | Pull one table for a data request | `clone { point_in_time }` + extract | | 5 | Copy of current state for debugging | `clone`, no `point_in_time` | | 6 | Lower-env refresh | `clone` — but not for OpenBao, it'd copy prod secrets down | ## Agreed approach and implementation plan ### Two restore types 1. **`clone { point_in_time }` → new instance** — restores to a point in time into a **new** instance. Covers the majority of real cases. 2. **In-place (`restore_backup_context`) → existing instance** — destructive, last resort only. ### Starting with type 1 We are starting with **type 1 (`clone { point_in_time }`)** since it covers most use cases. The current `postgresql@28.2.0` module has no clone/restore input, so a raw `google_sql_database_instance` resource with a `clone { point_in_time }` block will be added to the provisioner. The MR will be reviewed by the Runway team. Once merged, an end-to-end restore drill in staging will be run to record the RTO. Scenarios covered in this phase: | # | Case | Terraform | Cutover | |----|------|-----------|---------| | 1 | Bad migration or deploy; whole DB bad, nothing since worth keeping | `clone { point_in_time }` → new instance + cutover. `restore_backup_context` in place only as a last resort | Full | | 2 | Accidental `DELETE`/`UPDATE`; rest of the DB is fine | `clone { point_in_time }` → side instance → extract | None | | 3 | Accidental `DROP TABLE` | `clone { point_in_time }` → side instance → extract | None | | 5 | `DROP DATABASE` / mass loss | `clone { point_in_time }` → new instance + cutover. `restore_backup_context` in place only as a last resort | Full | | 6 | Storage or index corruption | `clone { point_in_time }` to establish when it started, then new instance + cutover | Full | ## Updated Proposed plan (2026-08-28) ### 1. Test and validate the clone + import process Clone → import → apply → connect → extract → remove the entry, in staging. Record the RTO. Gaps identified so far, to be covered by that testing: - [ ] **Authentication targets the right project.** `cloudsql-dr:authenticate` hardcodes `VAULT_GCP_IMPERSONATED_ACCOUNT="gitlab-runway-production--provisioner"` and never derives it from `ENVIRONMENT`, so a staging run would mint a production token. - [ ] **The clone job runs green.** Per !1815 it has never completed successfully. - [ ] **`managed.yml` accepts `import`.** `managed.schema.json` sets `additionalProperties: false` and has no such property, so the entry fails `json:schema` CI today. - [ ] **The post-import plan is import plus in-place only** — never destroy/recreate, which would wipe the restored data. - [ ] **We can reach and read the clone** once Terraform has wired PSC, DNS and IAM — the blocker from Test 2. - [ ] **Teardown actually destroys the clone.** `prevent_db_deletion` defaults to `true` and feeds both deletion-protection flags, so removing the entry may fail. - [ ] **Clone naming fits.** Names are capped at 33 characters because they are embedded in PSC endpoint names, and `runway-db-secrets-manager-gke` is already 29. - [ ] **`retain_backups_on_delete`** is unset and defaults to `false`, so a deleted instance takes its backups with it. ### 2. Test in-place restore The last-resort path, `gcloud sql backups restore` against an existing instance. Needs testing on its own because it is the one path that does not go through the clone flow. - [ ] **Confirm the data-loss window** behaves as documented — everything written since the backup is gone. - [ ] **Confirm backup configuration is reset** to Google defaults afterwards, and whether PITR ends up switched off. - [ ] **Confirm Terraform reconciliation** — an ordinary Atlantis plan/apply should show the `backup_configuration` drift and put it back. - [ ] **Confirm PITR is usable again** afterwards, otherwise the clone path above stops working. ### 3. Cutover testing Keep the imported clone and repoint the workload to it. - [ ] **Repoint the workload** to the clone's `instance_connection_name` and confirm it comes up healthy. - [ ] **OpenBao unseals from the restored database.** - [ ] **Observability follows the new instance.** Service catalog and metrics catalog entries live in the runbooks repo and are keyed on instance name, so Terraform cannot create them — confirm what has to be pre-staged. - [ ] **Decommissioning the old instance** is clean, including what happens to its backups and PSC endpoint. ### 4. Update the documentation and runbook Write it against the procedure steps that were executed during testing, not against the design. ## DRI @rhenchen.gitlab and @amritasinha