ops(runbook): tested PITR procedure for production schema rollback (ADR-016 follow-up)

xref:adrs/adr-016-forward-only-migrations.adoc[ADR-016] makes PITR (point-in-time recovery via pg_basebackup + WAL replay) the production rollback path for schema regressions. The forward-only stance is only fully load-bearing in prod once a step-by-step PITR runbook exists and has been tested at least once.

Acceptance criteria

  1. Operations runbook page (e.g. docs/modules/ROOT/pages/runbooks/pitr.adoc or under an existing operations docs structure) covering:
    • What signals indicate "we need PITR" vs. "forward-fix migration."
    • Pre-PITR checklist (snapshot current state, identify recovery target time, communicate with on-call).
    • Step-by-step PITR procedure for a single canopy service database.
    • Step-by-step PITR procedure for the cross-service case (every program DB rolled back to the same wall-clock moment).
    • Post-recovery validation: confirm hash chains intact (/v1/security/fti/chain-status, /v1/security/audit/chain-status if exposed), confirm audit_events and fti_audit_log integrity, verify no signed determinations were truncated.
  2. The runbook has actually been executed against a non-prod environment at least once. The test run is documented (date, operator, recovery target, outcome) so future on-call has a known-good reference point.
  3. Cross-link from xref:adrs/adr-016-forward-only-migrations.adoc[ADR-016] Negative-consequences section.

Why this isn't blocking ADR-016

Forward-only migrations are a coding policy: every contributor follows the expand-contract pattern. PITR is the operational escape hatch for the rare cases where a forward-fix migration isn't enough (data corruption, destructive forward migration that landed in prod). The two are independent.

If we ship ADR-016 without this runbook, contributors get the right policy and cargo xtask migrate rollback covers dev / CI. The gap is only that prod incident response would have to figure PITR out under fire — undesirable but not unsafe.

Source

ADR-016 Negative-consequences section explicitly flagged this as a separate deliverable.