This project is archived. Its data is read-only. This project is read-only.
Add native support for running `dbt test` as part of `meltano elt`
### Problem to solve As a fan of `dbt`, I really like the ability to run `dbt test` as part of my regular ELT pipelines. In fact, outside of Meltano I'd probably setup Airflow DAGs that would run a dbt project in stages like: 1. `dbt test` against the source rows that have been EL'd in 2. `dbt run` to build the analytics models 3. `dbt test` again to validate the analytics models If any of those tasks fail, the whole DAG would fail. However, there doesn't seem to be a way to actually run `dbt test` as part of a `meltano elt` pipeline, which means that I would have to refactor my project to use Meltano for "EL" but dbt and airflow separately for the rest... So overall, the problem is that: **I want meltano to support running dbt tests in my pipelines so I can be confident about the quality of my pipeline runs without configuring a separate dbt project**. ### Target audience Data engineers who want to use `dbt test` to automate schema and data tests for their pipelines. ### Proposal My initial idea would maybe be to extend `meltano elt` with another argument, like... ``` meltano elt tap-gitlab target-postgres --transform run --test ``` If the `transformer` supports it, that would add a "test" stage on the end of the pipeline so that it would become tap > target > transform > test. ### What does success look like, and how can we measure that? Not sure; I'm sure the maintainers here would have a better idea of how to tackle this... For me, success would be that I could guarantee that `dbt test` runs every time the related tap runs :smile: ### Regression test (Ensure the feature doesn't cause any regressions) - [ ] Write adequate test cases and submit test results - [ ] Test results should be reviewed by a person from the team ### Links / references _Please note that this was taken from GitLab, to be changed accordingly_
issue