fix(indexer): complete a plan whose source table does not exist
What does this MR do and why?
On GitLab 19.3 and 19.4 the agent-session Siphon tables do not exist yet. Their plans fail on every run, so a schema upgrade never activates. Now a plan whose source table is missing completes with nothing indexed. The upgrade activates, and the indexing status leaves that entity out until the table appears. The plan backfills from the start once the table appears. The check is generic (ClickHouse UNKNOWN_TABLE) and needs no schema bump.
Err(PipelineError::SourceUnavailable(reason)) => {
self.complete_without_source(&checkpoint_key, synced_until, &reason).await
}
// complete_without_source
self.checkpoint_store
.consolidate(checkpoint_key, &synced_until.unwrap_or(DateTime::<Utc>::UNIX_EPOCH))
.awaitRelated Issues
Relates to #1130 (closed).
Testing
A YAML scenario drops the table, runs, recreates it, and checks the backfill. A migration test checks that the upgrade activates with the table missing, and a negative test checks that a missing graph table still fails. On a VM, an upgrade from 0.150.0 activated in 76 s and the backfill landed 20 s after the table appeared.
Performance Analysis
- This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Details
- Status: a plan whose checkpoint has
indexed_atand the epoch watermark is left out of the status response, the counts, and the ready computation. No proto change. - A plan that indexed rows before its table vanished keeps its watermark.
- Why runtime detection, not a schema gated on the Rails version: one image serves mixed Rails versions. A table that arrives later is picked up on the next run.