Add built-in support for new FAST_SYNC spec (aka BATCH message type)
This enhancement would add framework support for the new FAST_SYNC
spec as described on the meltano thread (meltano#2364 (closed)).
To kick off the discussion, what about this as a strawman spec:
List of spec changes to support Fast Sync (partial, wip):
-
register_batch_export_handler()
- Registers a handler function to respond to batch export requests. Includes in the registration command a declaration of what file type and storage options are supported by the handler, along with the relative priority of the specific handler. -
register_batch_import_handler()
- Same as above but for targets.
Following from other design practices, we would not require that the tap author knows how to implement the BATCH message type, just that they return file paths in a way we can properly pass them to the downstream client (according to spec work on meltano#2364 (closed)).
Example:
In the case of a Redshift UNLOAD command, the register_batch_export_handler()
might give a function to execute the UNLOAD command, save to S3, and then download the files locally and return the corresponding local filepaths.