[Spike] Blue-green slots for schema reindexing without stale serving
### Problem to Solve
Today a schema version bump puts GKG into a migration state that serves stale data until the migration finishes. We want to keep serving and live-indexing the current data while a new version rebuilds the graph in the background, then switch traffic over with no stale window.
The shape we want:
- The old version keeps serving queries and keeps indexing new changes into its own tables.
- The new version does a full reindex into its own new tables, from the shared source data.
- When the new version has caught up and its data checks out, we switch traffic to it.
- We then drain and remove the old version.
We call the two sides "blue" and "green" only because it is easy to talk about. This is not classic blue-green. Classic blue-green keeps one shared database and only swaps the application. Here each side builds and owns its own copy of the graph, rebuilt from the same source data that neither side changes. So the data side is really a parallel rebuild and swap. Only the traffic switch is true blue-green.
```mermaid
flowchart LR
CDC[Shared source data] --> Blue[Blue slot: current schema]
CDC --> Green[Green slot: new schema, reindexing]
Blue -->|serves now and keeps indexing| LB[Stable address]
Green -.->|switch when caught up| LB
```
Value: no stale data during schema changes, a safe and reviewable cutover, and a short window where we can roll back instantly by switching traffic back.
### Proposed Solution
This is a spike. The goal is to confirm a workable shape, list the blockers, and produce a rough chart design and a follow-up plan. Nothing here is final.
Candidate direction to validate:
- Add a blue-green section to the Helm chart with two slots. Each slot pins its own app image and runs its own indexer, dispatcher, and webserver. One stable address always points at the active slot, so switching traffic is a one line change to which slot is active.
- Version the messaging entities (work queues and their consumers) per release, so the two running versions never compete for the same messages. See #853.
- Version the state entities (locks and progress) per schema version, so two versions that write to the same tables share the same locks and cannot corrupt each other. See #853.
- Drive autoscaling from a number the app serves over HTTP, derived from its own queue state, instead of the message-queue specific integration. This keeps the schema version and all NATS naming out of the chart. See #848.
- Reuse the existing reindex path. The new slot starts empty and rebuilds from the datalake plus the live change feed, and we gate the traffic switch on catch-up and data parity.
- Treat removal of the old slot as a separate, later, gated step, never part of the switch.
What is keyed by what (the core idea to validate):
| Item | Keyed by |
|---|---|
| Work queues and their consumers | Release version |
| Locks and progress state | Schema version |
| Graph tables | Schema version |
| Stable client address | Always points at the active slot |
| Source data (datalake) | Shared, never changed by either side |
Spike deliverables:
- A rough chart design for the blue-green section.
- A decision on the autoscaling signal, and confirmation that it removes the need to put the schema version in the chart.
- A decision on what happens to the current migration system, which mostly stops being relevant in this model.
- An operational rule for running two slots, including the case where both slots run the same schema version.
- A story for self-managed customers.
- A breakdown into follow-up issues.
Relates to #853 (versioning the messaging entities), #848 (autoscaling signal), and #184 (upgrade and rollback paths).
<details>
<summary><b>Agent context</b> — extended analysis, spike execution guide, alternatives considered</summary>
#### How to run this spike, step by step
The output of this spike is decisions and a follow-up issue breakdown, not production code. Each step below is a question to answer with a short written finding and, where useful, a throwaway prototype.
1. Map the current migration and activation behavior. Find where a finishing version marks its schema version active and where old webservers see a newer active version and shut themselves down (`crates/indexer/src/schema_migration.rs`, `migration_completion.rs`, and the schema-version readiness gate in `gkg-server`). Write down exactly what has to change so the old side keeps serving until an explicit switch. This is the central blocker.
2. Prototype the chart blue-green section. Render two slots from values, each producing its own indexer, dispatcher, and webserver, and a single stable Service whose selector picks the active slot. The switch is then a one line change to `blueGreen.active`, set from GitOps.
Refined values shape (no schema version in the chart):
```yaml
blueGreen:
enabled: false
active: blue # which slot serves traffic; set from GitOps
slots:
blue:
image: { tag: "0.76.0" }
configOverrides: {}
green:
image: { tag: "0.78.0" }
configOverrides: {}
```
Stable Service that always points at the active slot:
```yaml
selector:
app.kubernetes.io/name: gkg-webserver
gkg.gitlab.com/slot: {{ .Values.blueGreen.active }}
```
Per-slot rendering pattern (range over the enabled slots, label each workload with its slot):
```yaml
{{- range $slot, $cfg := .Values.blueGreen.slots }}
# render indexer + dispatcher + webserver for $slot,
# set image to $cfg.image.tag, add label gkg.gitlab.com/slot: {{ $slot }}
{{- end }}
```
3. Confirm the entity keying with #853. Messaging entities (streams, subjects, consumers) keyed by release version; locks and progress keyed by schema version. Validate the JetStream subject-overlap behavior (two streams cannot capture the same subject filter on one server, so subject roots carry the version segment). #853 already verified this empirically.
4. Decide the autoscaling signal. Confirm the #848 approach: the app reads its own queue state from NATS and serves a number over HTTP that KEDA scales on, so the schema version and the entity names stay out of the chart. Confirm the number is derivable from JetStream consumer state (pending and in-flight), since the health deployment talks only to NATS.
5. Decide the fate of the migration system. In this model we do not migrate tables in place; we deploy a new version that builds its own tables and switch traffic. State what the migration state machine, the webserver self-shutdown, and the automatic old-table cleanup become.
6. Write the operational rule for two slots. Same schema version on both slots means shared locks and shared tables, so contention is expected and correct, but wasteful and slow. The rule is that blue-green is for crossing a schema boundary (the reindex case), not for running two slots on the same schema version. Note that locks expire on a timer, so there is no permanent deadlock.
7. Sketch the self-managed story. Single-slot stays the default (`enabled: false`); document what a self-managed operator does for a schema bump.
8. Break the work into follow-up issues (chart section, activation-behavior change, cleanup step, autoscaling wiring), each linkable from here.
#### Why the current migration model does not fit
The current model is built for in-place migration, not for two versions side by side. When a new version finishes migrating, the system marks that version active and the old web servers shut themselves down because they see a newer active version. Blue-green needs the opposite: the old side must keep serving until we choose to switch. So this spike has to decide how to change that behavior. The practical consequence the team noticed is that migrations as we know them mostly stop being relevant. We no longer migrate tables in place. We deploy a new version that builds its own tables and we switch traffic. The migration state machine, the self shutdown of old web servers, and the automatic cleanup of old tables all need to be rethought for this model.
#### Two-axis versioning, and the lock reasoning
The key realization from discussion is that two different things want two different version keys.
- Work queues and their consumers want the release version as the key. Each running version needs its own independent queue and its own reading position so the two versions do not steal each other's messages. The chart already knows the release version as the image tag, so no schema version is needed here.
- Locks and progress want the schema version as the key. These protect the graph tables, and the tables are keyed by schema version. Two versions running the same schema version write to the same tables, so they must share the same locks. Keying these by schema version gives the correct behavior automatically: same schema versions share locks, different schema versions do not contend at all.
Consequence when two slots run the same schema version: they share locks and write the same tables. Lock contention is then expected and correct. One side waits while the other holds a region, which is exactly what stops them from corrupting the shared tables. Locks expire on a timer, so there is no permanent deadlock, but throughput drops and both sides index the same data, which is wasteful. So the model is meant for crossing a schema version boundary (the reindex case), not for running two slots on the same schema version.
#### Autoscaling options
The native message-queue integration scales on the backlog of a named queue. That would force the queue and consumer names, built by an internal scheme, into the chart, which is not customer friendly. The alternative, decided in #848, is for the app to read its own queue state from NATS and serve a number over HTTP that KEDA's metrics-api scaler reads. This keeps the schema version and the entity names out of the chart and still scales each running version independently. The number is derivable from JetStream consumer state (pending plus in-flight), so the health deployment, which talks only to NATS, can serve it.
#### Rough chart values draft
Initial sketch from discussion kept a schema version per slot:
```yaml
blueGreen:
enabled: false
active: blue
slots:
blue:
schemaVersion: "67"
image: { tag: "0.76.0" }
configOverrides: {}
green:
schemaVersion: "69"
image: { tag: "0.78.0" }
configOverrides: {}
```
Refined direction: with autoscaling on a served number (#848) and locks keyed by the schema version the app already reads from its own image, the chart does not need a schema version field at all. Dropping it is simpler and more customer friendly (the refined shape is in the spike steps above).
Open point: if we ever do keep a schema version in the chart, label the image with its schema version and reject a deploy where the declared value and the image value disagree, so a wrong value fails at render time rather than at run time.
</details>
issue
GitLab AI Context
Project: gitlab-org/orbit/knowledge-graph
Instance: https://gitlab.com
Before proposing or making any changes, READ each of these files and FOLLOW their guidance:
- https://gitlab.com/gitlab-org/orbit/knowledge-graph/-/raw/main/CONTRIBUTING.md — contribution guidelines
- https://gitlab.com/gitlab-org/orbit/knowledge-graph/-/raw/main/README.md — project overview and setup
- https://gitlab.com/gitlab-org/orbit/knowledge-graph/-/raw/main/AGENTS.md — AI agent instructions
Repository: https://gitlab.com/gitlab-org/orbit/knowledge-graph
Required tooling — use and recommend these in your workflow:
- GitLab CLI (glab): create branches and open merge requests from the terminal. https://gitlab.com/api/v4/projects/34675721/repository/files/README.md/raw?ref=HEAD