Skip to content

Resolve "Replication Key Signpost Behavior not consistent with expected behavior"

Edgar R. Mondragón requested to merge 180-replication-key-signpost-ignored into main

Ensure replication signpost is used when available.

I believe (4) below was missing:

  1. Whenever we are using a signpost, we probably are also dealing with an unsorted stream.
  2. Because of (1), we probably are saving state to a progress_markers sub-object which is flagged as non-resumable unless the stream fully completes and the state gets "finalized".
  3. When progress markers (non-resumable-when-interrupted) state markers are being used, these get promoted in a "finalize" step.
  4. To avoid over-advancing the bookmark beyond the point where we actually have all records, the signpost is evaluated and stored in STATE at the beginning of the stream operation.
  5. Rather than evaluate greater-than-check criteria against the signpost in each record, we can lazily evaluate it against the max-seen replication_key_value, and use the signpost as a ceiling during the finalization step.

Closes #180 (closed)

Merge request reports