Improving dbt plugin experience
We've talked recently about ways to improve the Meltano dbt plugin experience to make it more appealing vs running a standalone dbt project. Here are some thoughts (not all mine, just summarizing ideas I've heard):
- staging job code generation - we know quite a bit about the data being loaded if it was managed by Singer. We could have utilities to generate staging tables as accelerators or templates (meaning user intervention is needed).
- source.yml generation - auto configure source yamls
- create a higher level dag so we can run a command similar to the
+
syntax in dbt (dbt run --models +model_name
) where all dependencies are also rebuilt but extending into the Singer EL layer. - switching environments - Similar to how in the squared repo we use the environments feature to make it easy to switch between dbt environments. This might already be supported but just needs documentation on how best to use it. Or theres a way to make it even smoother.
- Others?
Let me know if this was already summarised somewhere else but I couldnt find an issue.