fix(indexer): support automatic schema rollback (re-activate or rebuild)

What does this MR do and why?

Rolling back to a previous schema version by deploying an older binary used to delete that version's tables, which made rollback impossible. Now redeploying an older binary rolls back automatically: if the target version's tables still exist it re-activates them instantly, and if they were already cleaned up it rebuilds and re-indexes them, promoting once complete. The version cleanup also now protects every in-migration version so a rebuild can't be dropped mid-flight.

Rollback decision matrix

Deployed binary vs. active version Target version's tables What happens
Same as active — No-op — already on this version
Newer than active — Forward migration: create tables, mark migrating, promote on completion
Older than active Still retained (within window) Case 1 — instant re-activation: mark the target active, retire the newer version. No re-index.
Older than active Already cleaned up (dropped) Case 2 — rebuild: recreate tables, re-index, promote on completion (after at least one indexer restart)
Older than a version that is mid-migration — Guarded — this binary won't promote a version it doesn't embed, so no crash-loop or outage; the in-flight higher version parks safely (surfaced by migrating_age_seconds). Deterministic abort of the interrupted migration is follow-up #958 (closed).

Closes #957 (closed). Follow-up: #958 (closed) (aborting a rollback that races an in-flight forward migration).

Testing

Unit and integration tests cover both rollback cases (direct re-activation and rebuild-then-promote), indexer readiness during a below-active rebuild, and the promoter guard (a dispatcher won't promote a migrating version it doesn't embed). Validated end-to-end on a local GDK stack: forward-migrate then roll back (Case 1), and double-migrate so the target's tables are garbage-collected, then roll back and rebuild (Case 2) — with live queries confirmed against the rolled-back version after each.

Performance Analysis

  • This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context — long-form analysis, file-by-file walkthroughs, alternatives considered

Origin. Production incident 2026-07-02: an operator rolled back active v76→v72 by redeploying the v72 binary, and v72's tables were dropped ~70s later. Root cause was that the dispatcher's migration path treated any version mismatch as a forward migration and marked the below-active version migrating, which stripped its retired GC protection so the cleanup task dropped it. The first iteration of this MR simply refused downgrades; per follow-up direction it now supports rollback as a real operation instead.

Rollback flow (crates/indexer/src/schema/migration.rs, run_rollback). When the dispatcher finds active > SCHEMA_VERSION it acquires the migration lock, re-reads (skips if another pod already rolled back), then branches on version_tables_exist (a system.tables lookup — authoritative, unlike status which can lag):

  • Case 1, tables retained — reactivate_version: mark the embedded version active, retire the previously-active version. No migrating phase, no re-index.
  • Case 2, tables GC'd — run_migration_locked (the forward-migration body, extracted so it reuses the same DDL-create + mark-migrating path): recreate tables, mark migrating. MigrationCompletionChecker promotes it once re-indexing covers the enabled namespaces.

GC keep-set (migration_completion.rs): the LIST_DEAD_VERSION_OBJECTS keep-set protects all migrating versions (the version > active restriction was removed). Required so a Case 2 rebuild sitting below the active version isn't dropped mid-rebuild.

Indexer readiness (version.rs, classify_readiness): reordered so a matching active/migrating version returns Ready before the active > embedded → Outdated check. Without this, the indexer rebuilding a below-active version (Case 2) exits as "outdated" and never backfills, so the rebuild could never reach the promotion gate. A genuinely outdated binary (nothing migrating its version) still fails fast. This gap was found by the end-to-end run, not the unit/integration tests.

Promoter guard (migration_completion.rs, check_completion): a dispatcher only promotes a migrating version it embeds (migrating == SCHEMA_VERSION). Without it, a straggler or rolled-back dispatcher whose completion checker runs while a higher version is mid-migration would promote that version and flip itself Outdated — a self-inflicted crash-loop that silently defeats the rollback. With the guard, such a version simply parks (surfaced by migrating_age_seconds). This makes the rollback-through-an-in-flight-migration case safe; deterministically aborting the in-flight higher version (and the mixed-version pointer-flapping it can cause) is deferred to #958 (closed).

Known first-boot behavior. The indexer and dispatcher boot independently, so the indexer can poll readiness before the dispatcher writes the migrating row; it exits Outdated and Kubernetes restarts it, and the next boot proceeds. A Case 2 rebuild therefore completes after at least one indexer restart, not on the first boot — a deliberate choice (keep fast detection of a truly wrong deploy), documented in schema_management.md.

Operational caveat. Any older-binary deploy triggers a rollback automatically — a stale image rolls back just like a deliberate target. Case 1 is an instant, reversible pointer swap within the retention window; Case 2 re-indexes from scratch. Documented in the design doc.

e2e validation. Both cases were driven against the local GDK ClickHouse (scoped to one enabled namespace so migrations complete). Case 1 and Case 2 both reached active with live queries returning real rows.

%{all_commits}

Edited by Jean-Gabriel Doyon

Merge request reports

Loading
Loading