## Problem Praefect stores authoritative repository state in a PostgreSQL database (replication jobs, generation numbers, storage assignments, etc.). When an operator takes a disk snapshot of a Gitaly node, the on-disk repository state and the Praefect database state can diverge: - A snapshot captures on-disk data at a point in time, but the Praefect DB may have already recorded writes that have not yet been flushed to disk, or vice-versa. - Restoring from such a snapshot can leave Praefect believing a replica is at a higher generation than the restored disk actually reflects. - This leads to silent data loss, incorrect primary election, or failed replication jobs. ## Proposal Design a first-class mechanism that allows operators to safely take disk snapshots of Gitaly storage nodes in a Praefect-managed cluster. The solution must ensure that, at the moment the snapshot is taken, the on-disk state is consistent with what the Praefect database believes to be true. Possible approaches to explore: 1. **Praefect quiesce API** — expose an RPC (or admin command) that temporarily pauses write traffic to a specific storage node and flushes/confirms outstanding replication jobs, allowing the operator to take a snapshot while the node is quiesced, then resume. 2. **Snapshot metadata export** — at snapshot time, export a Praefect DB checkpoint (e.g. generation numbers per repository) alongside the disk snapshot, and provide a restore procedure that reconciles the DB from that checkpoint. 3. **Coordinated snapshot hook** — provide a pre/post snapshot hook that Praefect is aware of, so it can mark the node as read-only, drain in-flight RPCs, and update the DB atomically. ## References - Praefect replication and generation tracking: `internal/praefect/` - Praefect database schema: `internal/praefect/datastore/`