Add support for ACTIVATE_VERSION message types
From the singer-python library:
ACTIVATE_VERSION message (EXPERIMENTAL). The ACTIVATE_VERSION messages has these fields: * stream - The name of the stream. * version - The version number to activate. This is a signal to the Target that it should delete all previously seen data and replace it with all the RECORDs it has seen where the record's version matches this version number. Note that this feature is experimental. Most Taps and Targets should not need to use the "version" field of "RECORD" messages or the "ACTIVATE_VERSION" message at all. msg = singer.ActivateVersionMessage( stream='users', version=2)
Implementing for taps:
I think we can safely implement for taps and send the message by default. For cases where targets cannot tolerate the unknown message types, we should support a disable_activate_version_messages=True
option.
FULL_TABLE
replication is selected in the tap:
When - Initialize a version number (likely an epoch-based integer): https://github.com/transferwise/pipelinewise-tap-snowflake/blob/aa89f2e4235999dbeafc7406a7f8b382542d8d5b/tap_snowflake/sync_strategies/common.py#L33
- Include
version
as property within emittedRECORD
messages. https://github.com/transferwise/pipelinewise-tap-snowflake/blob/aa89f2e4235999dbeafc7406a7f8b382542d8d5b/tap_snowflake/sync_strategies/common.py#L200 - Emit
ACTIVATE_VERSION
at the beginning of the firstFULL_TABLE
sync operation: https://github.com/transferwise/pipelinewise-tap-snowflake/blob/aa89f2e4235999dbeafc7406a7f8b382542d8d5b/tap_snowflake/sync_strategies/full_table.py#L87-L95 - Emit
ACTIVATE_VERSION
after a successfulFULL_TABLE
sync: https://github.com/transferwise/pipelinewise-tap-snowflake/blob/aa89f2e4235999dbeafc7406a7f8b382542d8d5b/tap_snowflake/sync_strategies/full_table.py#L114
Edited by AJ Steers