feat(indexer): two-axis NATS versioning with idle-release GC
What does this MR do and why?
Blue-green deployments run two GKG releases side by side, but every NATS entity was keyed by the schema version, so only schema bumps counted as a new version. This keys messaging entities by release version and state entities by schema version, isolating queues while same-schema releases still share locks. Retired releases' queues are cleaned up automatically: a starting process deletes streams that have shown no activity for a configurable window, with no extra coordination infrastructure.
Related Issues
Closes #853 (closed).
Testing
Unit tests cover two-axis name resolution, release-segment sanitization, and the managed-stream name parser. NATS-container integration tests cover the release GC sweep (deletes idle fixture streams, spares foreign streams), the schema-state cleanup, and no-op idempotency. Validated end-to-end against GDK: planted retired-release fixture streams were collected at startup, foreign and current-release streams survived, and the durable-consumer starvation case (below) was caught live and fixed. cargo clippy --workspace --all-features --tests, cargo fmt --check, and the indexer unit suite pass locally; the config JSON schema is regenerated in sync.
Performance Analysis
The GC is a single stream listing at process startup — no periodic tasks, no background loops, no hot-path impact.
- This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context — long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered
Two-axis versioning. NatsVersioner carries a release: String and a schema_version: u32. stream(), subject(), tag(), and resolve_stream_and_subject() key off the release; bucket() keys off the schema version. The release segment is the build version from gkg_utils::version::get(), sanitized to a NATS-safe token (SANITIZE_RELEASE_PATTERN collapses non-alphanumeric runs to -, e.g. 0.84.1 → 0-84-1) because it appears as a dot-delimited subject token (v0-84-1.sdlc…, v0-84-1.dlq.>). This follows Bohdan's refinement on #853 (closed): messaging needs per-release isolation, while locks/progress must be shared by same-schema releases (they write the same graph tables) and independent across schemas. Answering the open question on #853 (closed): the release key is the application/build version, which changes per deployment generation and is identical across replicas of one image — replicas share a queue, distinct images are isolated.
Cleanup split. The old single-axis cleanup_version(u32) split by axis: cleanup_schema_state(schema_version) deletes the locking + progress KV buckets and stays hooked to the migration-completion GC (bucket lifecycle is decided by ClickHouse migration state, not NATS-observable activity). The release axis is handled by gc_idle_release_streams (below); an intermediate cleanup_release_messaging(release) API ended up with no production callers and was removed.
Registry-less release GC. gc_idle_release_streams in nats/versioning.rs runs once at startup of every indexer and dispatcher, right after the process creates its own streams. There is no liveness registry: a live release's global dispatcher publishes to its work queue every minute by design, so stream activity doubles as a heartbeat. The sweep lists streams via jetstream.streams(), considers only names matching GKG_INDEXER_V* / GKG_DEAD_LETTERS_V* (release_from_stream_name returns None for foreign streams, making them structurally unreachable), groups by release, and deletes a release's observed streams only if every one is idle: not the current release, created older than the threshold (closes the new-deployment startup race), state.last_timestamp older than the threshold, and consumer_count == 0. A listing error aborts before any deletion, since sweeping a partial survey could miss the stream carrying a live release's activity. Failures are logged and self-retry at the next startup. Alternatives considered and dropped: a heartbeat KV bucket + scheduled GC task (an extra shared unversioned entity, background heartbeat loops, cron config, and metrics plumbing — the earlier iteration of this MR); consumer delivered.last_active as the liveness signal (delivery-only, misses publish-side liveness and costs a per-stream consumer listing).
Durable-consumer starvation fix. Live GDK testing surfaced a hole: subscribe-path durable consumers persist after their process dies, so a retired release's consumer_count stayed > 0 forever and the veto spared its streams permanently. Fix: get_or_create_consumer now sets inactive_threshold from the existing nats.consumer_inactive_threshold_secs (previously applied only to Siphon dispatch consumers). A connected consumer's continuous fetch loop keeps it active indefinitely; only dead releases' consumers reap. If a consumer is reaped while its indexer is merely down and later recreated, WorkQueue retention redelivers all unacked messages, so nothing is lost.
Config. nats.release_gc_idle_threshold_secs (default 3600, clamped ≥ 600 so the created-guard window stays comfortably wider than a slow rollout's startup). config/schemas/config.schema.json regenerated via mise schema:generate. Docs: versioning + GC section in docs/design-documents/indexing/sdlc_indexing.md, config rows in the server-configuration runbook.
%{all_commits}