cells: ADR backup and restore for Disaster Recovery

- [ADR-002: One GCP Project per Cell](decisions/002_gcp_project_boundary.md)

- [ADR-003: One GKE Cluster per Cell](decisions/003_num_gke_clusters_per_cell.md)

- [ADR-004: One VPC per Cell, with Private Service Connect for internal communication between Cells](decisions/004_vpc_subnet_design.md)

- [ADR-005: Cells use Flexible Reference Architectures](decisions/005_flexible_reference_architectures.md)

- [ADR-006: Use Geo for Disaster Recovery](decisions/006_disaster_recovery_geo.md)

- [ADR-005: Cells use Flexible Reference Architectures](decisions/005_flexible_reference_architectures.md)[006_disaster_recovery_geo.md](decisions/006_disaster_recovery_geo.md)

- [ADR-007: Cells 1.0 for internal customers only (obsolete)](decisions/007_internal_customers.md)

- [ADR-008: Cluster wide unique database sequences](decisions/008_database_sequences.md)

- [ADR-009: Initial Cell Sizes](decisions/009_cell_initial_sizing.md)

- [ADR 021: Topology Service Claims In Database Transactions](decisions/021_claims_in_database_transaction.md)

- [ADR 022: Ring definition for Protocells](decisions/022_ring_definition_for_protocells.md)

- [ADR 023: Database Configuration for Cells](decisions/023_database_configuration.md)

- [ADR-024: Use Backup and Restore for Disaster Recovery](decisions/006_disaster_recovery_geo.md)

## Links

toc_hide: true

---

**Note:** This ADR reflects an earlier approach. The current decision is documented in [024_disaster_recovery_cells.md](024_disaster_recovery_cells.md)

## Context

We discussed whether we should use Geo for Disaster Recovery in [this issue](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/25246).

---

owning-stage: "~devops::tenant services" # because Tenant Scale is under this

title: 'Cells ADR 024: Use Backup and Restore for Disaster Recovery'

toc_hide: true

---

## Context

This ADR states the use of a backup and restore strategy as Cells disaster recovery model.

Today, we regularly back up cell data and store those backups in AWS Vault. As discussed in

[this issue](https://gitlab.com/gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team/-/work_items/483),

these backups are intended to be the primary mechanism for Disaster Recovery (DR) in a Cells-based architecture.

## Decision

We will use **backup and restore as the primary Disaster Recovery mechanism for Cells**.

Each cell is responsible for producing consistent backups of its stateful components (for example databases, object storage, and configuration state). In a disaster scenario,

a new cell can be provisioned potentially in the same or different AWS region and restored from the most recent valid backup.

This approach implies:

- Backups are treated as first-class recovery artifacts.

- Restore workflows must be automated, repeatable, and regularly exercised.

- Recovery is performed by creating a new cell rather than attempting in-place repair of a failed one.

Backup data stored in AWS will be considered the source of truth for disaster recovery operations.

## Consequences

### Positive

- Strong alignment with the Cells isolation model: failures and recoveries are scoped to individual cells

- Reduced operational complexity/cost compared to global synchronous replication as we have with geo.

- Flexibility to restore into a new region or account, helping mitigate AWS quota exhaustion and regional outages.

- Clear and testable recovery procedures that can be validated through drills.

### Negative

- Unlike Geo, RTO will be bounded by tenant provision and restore time which will be longer than a Geo failover.

- Unlike Geo, RPO will be bounded by the time of the last successful backup which is configurable and will be longer than a Geo failover.

- Restore workflows must be reliable; failures in restore tooling directly impact RTO.

### Operational Implications

- Backup freshness and integrity must be continuously monitored

- Restore procedures must be documented and automated.

- Regular disaster recovery exercises are required to ensure backups are usable under real conditions