Blue-green deployments: sequenced non-disruptive rollout
Problem to Solve
A schema change today reindexes in place and serves stale data until the migration finishes, and the switchover, the promotion of the new version, and the eviction of the old fleet all happen in one irreversible step. We want to keep serving and live-indexing the current version while a new version rebuilds the graph in the background, then switch traffic with no stale window and with instant rollback. Spike #877 (closed) confirmed the shape. This issue supersedes #877 (closed) with the ordered, non-disruptive implementation plan.
The constraint that shapes the plan: the running system must stay working at every step. Single-slot stays the default and the self-managed path. Blue-green is introduced behind switches that default to today's behavior. Green is stood up first running the same code and the current migration strategy, without receiving traffic. Only later, gated code changes enable the first real switch to a separately built slot.
flowchart LR
subgraph Today["Today: single fleet, in-place migration"]
S1[Shared source data] --> F1[One fleet]
F1 --> T1[Serves; migration flips in place, old pods self-exit]
end
subgraph Target["Target: blue-green"]
S2[Shared source data] --> B[Blue: current version, serving + live-indexing]
S2 --> G[Green: new version, rebuilding in background]
B -->|serves now| A[Stable address]
G -.->|switch when built + verified| A
endRelates to #853 (closed), #848 (closed), #580 (closed), #794, #443 (closed), #184 (closed), #957 (closed), #579 (closed). Supersedes #877 (closed).
Proposed Solution
Deliver blue-green as a strict sequence. Each item is shippable on its own and leaves the system in a working state. Every new behavior is gated by a switch that defaults to today's behavior, so work can stop at any item without disruption. The first real traffic switch happens only at Phase 4.
Guiding rules:
- One master switch, default off. Off renders and behaves exactly as today.
- Green appears first with the same version and the current migration strategy, not serving.
- The stable client address never moves. Only which slot it targets changes.
- Single-slot stays the default and the self-managed story.
Phase 0 — Reversible foundations (no behavior change)
- 1. Add the blue-green master switch to the chart, default off, rendering exactly today's single deployment. Prove it is a no-op against the current release.
- 2. Add a new version state, "built and verified but not yet serving", to the schema control record, ahead of any code that writes it. Purely additive.
- 3. Teach retention to protect that new state before anything uses it, so a freshly built slot can never be cleaned up.
Phase 1 — Green in place, same code, same migration, not serving
- 4. Render a second slot running the same version as the live one, isolated on its own messaging and progress state, receiving no traffic. Validate that two slots coexist.
- 5. Allow one coordinator per slot (only one is allowed cluster-wide today). Confirm cross-slot locking is safe and size lock lifetimes for a long full rebuild.
- 6. Make health reporting, autoscaling, disruption budgets, and metrics slot-aware, so each slot scales and reports independently.
- 7. Make messaging cleanup reap a retired slot's streams on teardown.
Phase 2 — Decouple serving from the global pointer (gated off)
- 8. Let a slot serve when its own data is built and verified, instead of only when it is the single global active version.
- 9. Stop the completion step from evicting the live slot when a new slot finishes. Under the switch, mark the new slot built-and-verified and leave the live slot serving. With the switch off, keep today's in-place promotion untouched.
- 10. Stop the live slot from shutting itself down when another slot reaches a higher version. Keep self-shutdown only for the single-slot rollback path.
- 11. Make the worker startup check slot-aware, so a live slot's workers survive a restart during an overlap.
Phase 3 — Safe-to-switch gate (correctness)
- 12. Add a data-completeness and parity gate that must pass before a new slot is marked serviceable, measured against a frozen source snapshot per top-level namespace. Closes the incomplete-promotion failure class (#794).
- 13. Confirm the shared source-side views are owned by a single actor and hold up while a slot does a full rebuild alongside live indexing (#579 (closed)).
Phase 4 — First switch (new version on green, blue keeps serving)
- 14. Run green on a new schema version, rebuilding into its own tables, while blue serves the current version.
- 15. Implement the traffic switch as a one-line active-slot change, and tune the existing connection-age drain so clients move to green within a bounded window. Confirm rollback is a revert of that one line.
- 16. Report a distinct deploying or degraded status during a rebuild instead of unhealthy, and fix the status check that currently flags every deploy as unhealthy (#580 (closed)).
Phase 5 — Teardown, rollback, self-managed, cleanup
- 17. Define gated old-slot teardown: switch, wait out the drain window, then retire and clean up. Bound shared-database, messaging, and node capacity during the overlap.
- 18. Define rollback by re-flipping to the still-warm previous slot within the retention window.
- 19. Keep single-slot the default and the self-managed path, and document the self-managed schema-bump story.
- 20. Retire the superseded parts of the old migration model and update the design docs.
Agent context — mechanics per item, research corrections, decisions, primitives to reuse
The primary blocker
gkg_schema_version.active (a single global u32) currently means three things at once: the traffic target, the only version any pod may serve, and the garbage-collection anchor. One write promotes green, evicts blue by SchemaWatcher self-exit, and switches traffic, irreversibly. Blue-green splits these three. Traffic moves to the K8s Service selector, serving keys off a new per-version status, and active is demoted to a bookkeeping/GC anchor. reactivate_version (the O(1) pointer swap) is kept only for single-slot rollback.
What already exists and must be reused, not rebuilt
- Per-version
vN_table prefixes pinned at webserver startup (crates/gkg-server/src/main.rs,crates/indexer/src/schema/version.rs). This is the data-isolation half of blue-green (!856 (merged)). - Two-axis NATS versioning: messaging keyed by release, locks and progress KV keyed by schema (
crates/indexer/src/nats/versioning.rs, !1985 (merged)/!1783 (merged), #853 (closed)). Two slots on different release images auto-isolate their streams, subjects, and consumers, so the chart needs no per-slot NATS naming. - App-served autoscaling signal plus per-pool KEDA metrics-api (
crates/health-check,chart/templates/indexer/scaledobject.yaml, #848 (closed)), already live in prod. The number is release-derived, so per-slot scaling needs no schema or NATS names in the chart. - Server-side connection draining already exists: tonic
max_connection_age_secs = 300andmax_connection_age_grace_secs = 30(config/default.yaml:150-151,crates/gkg-server/src/grpc/server.rs:109-110), commented "for L4 ILB rebalancing" (WI #330 (closed)). Both clients honor GOAWAY (see drain note below). - Completion checker and GC keep-set that always protects
migrating(crates/indexer/src/orchestrator/scheduled/migration_completion.rs). - The stable address is an L4 internal LoadBalancer republished via GCP PSC; the forwarding-rule name is pinned in Terraform (
config-mgmt/modules/orbit-environment/private-service-connect.tf, consumer side ingprd/gstg gkg-psc.tf).
Per-item mechanics
- 1. Gate on
blueGreen.enabled. Off keeps current resource names and selectors. On renders slot-suffixed net-new Deployments and gives the webserver Service selector a slot key. Note apps/v1 immutability:Deployment.spec.selectoris immutable so slotting requires net-new Deployment names (old ones pruned), butService.spec.selectoris mutable so the Service evolves in place. The Service must never be deleted or recreated, or GKE regenerates the forwarding-rule name and the pinned PSC hash breaks (Terraform apply plus re-plumbing). - 2.
gkg_schema_version.statusis a fixedEnum8('active'=1,'migrating'=2,'retired'=3,'dropped'=4)created withCREATE TABLE IF NOT EXISTS(a no-op on existing clusters), and all writers insert the status as a string literal. A new value (for example'standby'=5) needs an explicitALTER TABLE ... MODIFY COLUMN status Enum8(...)on the never-prefixed, cross-slot control table, sequenced in the dispatcher migration path before any pod writes it. - 3. Add the new status to the GC protected keep-set (
LIST_DEAD_VERSION_OBJECTS) in the same change that starts writing it. If the completion step flips green offmigratingto the new status before retention protects it, the next GC tick can drop green's freshly built tables. - 4. Same-schema two-slot is validation-only, not a serving mode. Two same-schema slots share
vN_tables and schema-keyed locks and double-index the same CDC, which #877 (closed) explicitly discourages; used here only to prove the chart renders two slots and NATS isolates by release. Green does not receive traffic (selector stays on blue). - 5. Drop the singleton dispatcher assumption (
chart/templates/dispatcher/deployment.yaml,replicas:1,Recreate). Theschema_migrationlock lives in the schema-versioned locks bucket, so different-schema slots never contend and same-schema slots correctly serialize. Locking is fire-and-forget TTL with no renewal (crates/indexer/src/locking.rs); a long green rebuild can outlive a TTL sized for steady state, with prod precedent for a redelivery rug-pull when TTL equaled ack-wait. Size and verify TTLs against the expected full-rebuild duration. - 6. Health-check auto-enrollment (
healthCheckDefaultDeploymentsin_helpers.tpl), the ScaledObject, the PDB, and the PodMonitor are all fullname/component-keyed. Decide early whether slots are two helm releases (naming isolates automatically) or one release with slot suffixes (all four templates gain a slot key). One release with a singleblueGreen.activevalue is the simpler control plane and matches the #877 (closed) values shape; two releases per environment means a four-way matrix across staging and prod plus a shared release owning the stable Service. - 7. Idle-release stream GC currently runs only in the dispatcher path, not the indexer path, despite the !1985 (merged) intent. Wire it so a retired slot's release streams are reaped at teardown.
- 8, 10, 11. Readiness classification keys on
active == embeddedtoday, andactive > embeddedtriggers self-exit (crates/gkg-server/src/schema_watcher.rs). Change: serve when the embedded version's status is active or the new built-and-verified status; do not self-exit the traffic-target slot (in the recommended modelactivenever climbs above blue's embedded during the switch); keep self-exit only for single-slot rollback. The indexer boot gate is one-shot and fatal on outdated (crates/indexer/src/schema/version.rs), so make it slot-scoped or a blue indexer pod reschedule during overlap crashloops. - 9. Gate the completion behavior on
blueGreen.enabled. With the switch on, the completion step marks green built-and-verified and does not retire blue, clear blue's campaign, or stop merges. With the switch off, today's auto-promote and self-exit are untouched, which is the single-slot and self-managed upgrade path. Gutting auto-promote globally would break every single-fleet upgrade. - 12. No cross-slot catch-up process is needed: green consumes the same source feed independently and live-indexes concurrently with its backfill, so it converges to source head on its own. The gate is a read-only completeness and parity check, not a running process. Today's completion is checkpoint-existence only, SDLC-only, no row counts (
migration_completion.rs), which is how #794 promoted at about 52 percent. Reference: freeze a start watermark per source table (the existing_siphon_watermark, #849 (closed), plus the checkpoint watermark already stored), compare green's per-table counts against the source snapshot at that frozen watermark, per top-level namespace, with ontology-derived tolerance (invariant tables near-exact, changed-semantics tables coverage-based), plus a cheap live-lag assertion. Owner: #443 (closed). - 13. Traversal-path views are datalake-side and unprefixed, read by both the completion gate and code backfill. With one dispatcher per slot, confirm the shared datalake schema and its compensating views are owned by a single actor, and confirm the #579 (closed) fix holds when a slot does a full backfill against continuously mutating shared tables.
- 15. The switch is a selector reswing via GitOps (flip
blueGreen.active). It moves only new connections; existing long-lived gRPC channels rebalance via the server GOAWAY on max-connection-age (bounded, about age plus grace, roughly 330s today). The one knob is gkg'smax_connection_age_secs; there is no client-side equivalent. Keep clients pointed at the ILB VIP, never per-pod DNS, orpick_firstreconnect can return to a surviving blue pod. Verify the actual workload-app ArgoCD syncPolicy (the prune and self-heal values that govern git-revert rollback come from theargocd-generic-servicechart default, not the outer wrapper) before relying on revert-to-rollback. - 16. The proto
ClusterStatusalready has aDegradedvariant but it is unreachable: the sidecar status is binary and the cluster status is a passthrough (crates/gkg-server/src/cluster_health.rs). Derive deploying/degraded from replica counts and per-slot schema state, count only the traffic-target slot against desired to fix #580 (closed), and report per-slot blocks. - 17. Teardown order: flip selector, wait at least max-connection-age plus grace so no client is mid-rebalance, then retire, GC, and stop merges, then Argo prune (prune is off, so teardown is explicit). Overlap bounds: cap green's autoscale ceiling while blue serves (shared ClickHouse has no cross-slot governor), and size the node pool for both fleets (the binding limit is local-SSD ephemeral storage, roughly five code pods per node at current requests on one 375 GiB SSD, so blue floor plus green ceiling needs a node-count bump).
- 18. Rollback within the retention window is a re-flip to the previous slot, which is still warm because teardown is a separate gated step. Reconcile with
reactivate_versionand the existing rollback path (!1984 (merged), #957 (closed)). - 20. Retire the superseded auto-promote and webserver self-shutdown as the primary mechanism once blue-green is the SaaS default, and update
docs/design-documents/schema_management.md,health_check.md,README.md,CONTEXT.md, and the crate map in the same change.
Open decisions for the roadmap owner
- Traffic authority: Service selector (recommended, no Terraform in the hot path, git-revert rollback) versus keeping
activeas the pointer flipped by a GitOps-triggered reactivation with a controller reconciling the selector. - Slot packaging: one release with slot suffixes versus two releases per environment.
- Parity reference and tolerance basis (frozen source snapshot recommended).
- Cutover drain window length via max-connection-age, traded against keeping blue warm for rollback.