fix(indexer): support automatic schema rollback (re-activate or rebuild)
What does this MR do and why?
Rolling back to a previous schema version by deploying an older binary used to delete that version's tables, which made rollback impossible. Now redeploying an older binary rolls back automatically: if the target version's tables still exist it re-activates them instantly, and if they were already cleaned up it rebuilds and re-indexes them, promoting once complete. The version cleanup also now protects every in-migration version so a rebuild can't be dropped mid-flight.
Rollback decision matrix
| Deployed binary vs. active version | Target version's tables | What happens |
|---|---|---|
| Same as active | — | No-op — already on this version |
| Newer than active | — | Forward migration: create tables, mark migrating, promote on completion |
| Older than active | Still retained (within window) | Case 1 — instant re-activation: mark the target active, retire the newer version. No re-index. |
| Older than active | Already cleaned up (dropped) | Case 2 — rebuild: recreate tables, re-index, promote on completion (after at least one indexer restart) |
| Older than a version that is mid-migration | — | Guarded — this binary won't promote a version it doesn't embed, so no crash-loop or outage; the in-flight higher version parks safely (surfaced by migrating_age_seconds). Deterministic abort of the interrupted migration is follow-up #958 (closed). |
Related Issues
Closes #957 (closed). Follow-up: #958 (closed) (aborting a rollback that races an in-flight forward migration).
Testing
Unit and integration tests cover both rollback cases (direct re-activation and rebuild-then-promote), indexer readiness during a below-active rebuild, and the promoter guard (a dispatcher won't promote a migrating version it doesn't embed). Validated end-to-end on a local GDK stack: forward-migrate then roll back (Case 1), and double-migrate so the target's tables are garbage-collected, then roll back and rebuild (Case 2) — with live queries confirmed against the rolled-back version after each.
Performance Analysis
- This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context — long-form analysis, file-by-file walkthroughs, alternatives considered
Origin. Production incident 2026-07-02: an operator rolled back active v76→v72 by redeploying the v72 binary, and v72's tables were dropped ~70s later. Root cause was that the dispatcher's migration path treated any version mismatch as a forward migration and marked the below-active version migrating, which stripped its retired GC protection so the cleanup task dropped it. The first iteration of this MR simply refused downgrades; per follow-up direction it now supports rollback as a real operation instead.
Rollback flow (crates/indexer/src/schema/migration.rs, run_rollback). When the dispatcher finds active > SCHEMA_VERSION it acquires the migration lock, re-reads (skips if another pod already rolled back), then branches on version_tables_exist (a system.tables lookup — authoritative, unlike status which can lag):
- Case 1, tables retained —
reactivate_version: mark the embedded versionactive, retire the previously-active version. Nomigratingphase, no re-index. - Case 2, tables GC'd —
run_migration_locked(the forward-migration body, extracted so it reuses the same DDL-create + mark-migrating path): recreate tables, markmigrating.MigrationCompletionCheckerpromotes it once re-indexing covers the enabled namespaces.
GC keep-set (migration_completion.rs): the LIST_DEAD_VERSION_OBJECTS keep-set protects all migrating versions (the version > active restriction was removed). Required so a Case 2 rebuild sitting below the active version isn't dropped mid-rebuild.
Indexer readiness (version.rs, classify_readiness): reordered so a matching active/migrating version returns Ready before the active > embedded → Outdated check. Without this, the indexer rebuilding a below-active version (Case 2) exits as "outdated" and never backfills, so the rebuild could never reach the promotion gate. A genuinely outdated binary (nothing migrating its version) still fails fast. This gap was found by the end-to-end run, not the unit/integration tests.
Promoter guard (migration_completion.rs, check_completion): a dispatcher only promotes a migrating version it embeds (migrating == SCHEMA_VERSION). Without it, a straggler or rolled-back dispatcher whose completion checker runs while a higher version is mid-migration would promote that version and flip itself Outdated — a self-inflicted crash-loop that silently defeats the rollback. With the guard, such a version simply parks (surfaced by migrating_age_seconds). This makes the rollback-through-an-in-flight-migration case safe; deterministically aborting the in-flight higher version (and the mixed-version pointer-flapping it can cause) is deferred to #958 (closed).
Known first-boot behavior. The indexer and dispatcher boot independently, so the indexer can poll readiness before the dispatcher writes the migrating row; it exits Outdated and Kubernetes restarts it, and the next boot proceeds. A Case 2 rebuild therefore completes after at least one indexer restart, not on the first boot — a deliberate choice (keep fast detection of a truly wrong deploy), documented in schema_management.md.
Operational caveat. Any older-binary deploy triggers a rollback automatically — a stale image rolls back just like a deliberate target. Case 1 is an instant, reversible pointer swap within the retention window; Case 2 re-indexes from scratch. Documented in the design doc.
e2e validation. Both cases were driven against the local GDK ClickHouse (scoped to one enabled namespace so migrations complete). Case 1 and Case 2 both reached active with live queries returning real rows.
%{all_commits}