Failure to run Custom Transforms when default Transform is not defined
What is the current bug behavior?
We can not run meltano elt ... transform run
if a default transform for a tap is not defined.
This problem has not been encountered until now as all taps that extract data from APIs can be bundled with default transforms.
But when using a tap with no static schema / predefined entities, we can not have default transforms.
That happens when trying to load data from a CSV file (tap-csv
) or extract data from a Postgres Database (tap-postgres
) or any other data source that we'll support in the future.
For those cases, we can not have default transforms, so there can not be a default dbt-tap-csv
project prepackaged with Meltano.
As an example of the bug encountered, if a user runs:
$ meltano elt tap-csv target-postgres --transform run
... ... ...
Extract & load complete!
Transform 'tap-csv' is missing, trying to install it...
Error: transform 'tap-csv' is not supported
Aborted!
What is the expected correct behavior?
We want to be able to support taps that extract from data sources or load CSV and other types of data files.
For those cases, we should be able to run the transform part of the ELT pipeline even if there is no default transform.
If there are custom transforms in the project's transform/models/my_meltano_project/
folder, those transforms should run successfully.
Steps to reproduce
$ meltano elt tap-csv target-postgres --transform run
... ... ...
Extract & load complete!
Transform 'tap-csv' is missing, trying to install it...
Error: transform 'tap-csv' is not supported
Aborted!
Possible fixes
We could add a dbt-tap-XXX
project with no transforms for each of those taps, but this seems more like a bandage than a proper fix.
We should update the way we execute transforms in order to support the case where no default transforms are available.
Further regression test
- Add a test for running
meltano elt ... --transform run
for a tap with no default transforms defined.tap-csv
should be a proper use case.