Exploration: Revamp Meltano Transforms
There is a lengthy discussion on #562 (closed) on adding full support for running various types of transformations using Meltano.
-
@iroussos complete opportunity assessment: https://docs.google.com/document/d/1bN4I-j9LmvAgdJuP8rhs1j0FpR5lE_rOmB6UKCNi-aQ/edit
From that issue:
I think that the way to solve all of the aforementioned issues should be one of the following:
-
Add an option for users to add custom packages and select which models are run (instead of only running the default package and what's inside the
my_meltano_project
folder).I would couple this update with also allowing users to only run transforms by using
meltano transform
:-
meltano transform XXX
==meltano elt XXX YYY --transform only
-
meltano transform MODEL_1 MODEL_2
--> run those models only (packages or whatever else a user wants) -
meltano add custom transform git_uri
--> Add package topackages.yml
and update yourdbt_project.yml
so that the model is available to be run.
-
-
(Power user feature) Add a
meltano dbt
ormeltano dbt_run
option that will allow users to directly access the fulldbt
functionality.It will allow power users to run dbt directly from Meltano's project root, without worrying about where their profile or the dbt installation is located.
Just add your package to
packages.yml
and update yourdbt_project.yml
, run the command you want and Meltano will take care of the rest.meltano dbt_run --models my_first_model my_second_model
equals in this case to going to the correct directory and running:dbt run --models my_first_model my_second_model --profiles-dir project_root/transform/profile/ --profile meltano