Add `meltano run [<plugin>...]` to run arbitrary plugins in a pipeline
As meltano run evolves we may opt to expand/change the invocation syntax but
a simple first iteration will be to focus on just the chain use case: we do a bit of work to validate blocks, but otherwise we run left to right and find ExtractLoad Blocks as appropriate. @tayloramurphy
It's simply meltano run tap-gitlab target-jsonl tap-gitlab target-csv dbt:run
with blocks run in series (with ExtractLoad blocks parsed and linked as needed).
Other sample valid invocations:
meltano run tap-gitlab map-remove-nulls target-msyql dbt:run
meltano run tap-gitlab target-msyql dbt:run tap-msqyl target-json
meltano run tap-gitlab target-msyql dbt:run dbt:test superset:build
Invalid invocations:
- command block between IOBlocks:
meltano run tap-gitlab dbt:run target-msyql
- starting IOBlock (tap-peloton) with no ending IOBlock (a target):
meltano run tap-peloton tap-gitlab target-mysql
- ending IOBlock (target-jsonl) with no starting IOBlock (a tap):
meltano run tap-gitlab target-mysql target-jsonl
- tap/target used as command block:
meltano run tap-gitlab:discovery
Job ID generation and STATE support:
- If no environment is provided: job ID is not generated and STATE is not supported.
- If environment is provided: job ID is automatically generated with a format such as
{environment_name}:{tap_name}-to-{target_name}
.
Explicitly out of scope in the first version:
- Running taps/targets as commands is out of scope at least in this first version.
- Retries, permissive or selective failures is out of scope at least in this first version. A failure at any point halts execution and no further blocks are executed.
Fully out of scope:
- There's no plan as of now to support individual block-level arg passing - and it will likely not be available in future revisions. For instance, this will not be possible:
meltano run dbt:run dbt:test[--verbose]
. Instead you'd need to usemeltano invoke
or create a customtest-verbose
command and execute like thismeltano run dbt:run dbt:test-verbose
.
Original issue body
As in:
-
meltano elt <extractor> <loader> [--transform=skip]
->meltano run <extractor> <loader>
, e.g.meltano run tap-foo target-bar
-
meltano elt <extractor> <loader> --transform=run
->meltano run <extractor> <loader> <transformer>
, e.g.meltano run tap-foo target-bar dbt
-
meltano elt <extractor> <loader> --transform=only
->meltano run <transformer> --with <extractor> --with <loader>
, e.g.meltano run dbt --with tap-foo target-bar
(see #2546 (closed))
And instead of `meltano schedule gitlab-to-jsonl tap-gitlab target-jsonl @hourly --transform=run:
schedules:
- name: gitlab-to-jsonl
extractor: tap-gitlab
loader: target-jsonl
transform: run
interval: '@hourly'
We get meltano schedule gitlab-to-jsonl tap-gitlab target-jsonl dbt @hourly
:
- name: gitlab-to-jsonl
interval: '@hourly'
pipeline: # or `run`?
- tap-gitlab
- target-jsonl
- dbt