Add native support for running `dbt test` as part of `meltano elt`
Problem to solve
As a fan of dbt
, I really like the ability to run dbt test
as part of my regular ELT pipelines. In fact, outside of Meltano I'd probably setup Airflow DAGs that would run a dbt project in stages like:
-
dbt test
against the source rows that have been EL'd in -
dbt run
to build the analytics models -
dbt test
again to validate the analytics models
If any of those tasks fail, the whole DAG would fail. However, there doesn't seem to be a way to actually run dbt test
as part of a meltano elt
pipeline, which means that I would have to refactor my project to use Meltano for "EL" but dbt and airflow separately for the rest...
So overall, the problem is that: I want meltano to support running dbt tests in my pipelines so I can be confident about the quality of my pipeline runs without configuring a separate dbt project.
Target audience
Data engineers who want to use dbt test
to automate schema and data tests for their pipelines.
Proposal
My initial idea would maybe be to extend meltano elt
with another argument, like...
meltano elt tap-gitlab target-postgres --transform run --test
If the transformer
supports it, that would add a "test" stage on the end of the pipeline so that it would become tap > target > transform > test.
What does success look like, and how can we measure that?
Not sure; I'm sure the maintainers here would have a better idea of how to tackle this...
For me, success would be that I could guarantee that dbt test
runs every time the related tap runs
Regression test
(Ensure the feature doesn't cause any regressions)
-
Write adequate test cases and submit test results -
Test results should be reviewed by a person from the team
Links / references
Please note that this was taken from GitLab, to be changed accordingly