Support incremental jobs for `meltano run`
Spun out from #3094 (closed).
This items are necessary for incremental jobs.
- Job ID support via
--job-id-prefix
xor--job-id-name-pattern
- May be sourced based on the environment name - or via a CLI flag for invocations which do not have
--environment
set. (#2301 (comment 750941505))
- May be sourced based on the environment name - or via a CLI flag for invocations which do not have
- Top-level
--no-state-update
to not save state. (!2432 (comment 757893658)) - Top-level
--full-refresh
to ignore saved state (which forces a full refresh). (!2432 (comment 757893658))
Of these three, I think the first is highest priority, while the second and third could optionally be delayed. In special circumstances where advanced control over state is needed, users can still fall back to meltano elt
to get those functions.
Proposed syntax:
Job ID:
Set it at the front of a run or automatically get one based on the tasks run.
meltano run --job-id=<id> tap-gitlab...
State:
We could set state requirements right after run
. For many tools (dbt included) there is a concept of incremental vs full-refresh runs. Those commands could propagate down to the specific plugins. Default is an incremental run.
meltano run --no-state-update tap-gitab ...
meltano run --full-refresh tap-gitlab ...