Adding Transform endpoint
After trying different approaches and discussion with @iroussos I stelled on the following scenario for running dbt run
and passing the database connection values to it:
- The dbt main function is invoked as part of the
/transofrm/<model_name>
API endpoint request handler - The dbt main function is run from the
transform
folder which mimics the way it is run on the cli. - The connection configuration is specified in the
profiles.yml
file which are referencing the env variables. - Within the request, handler connection values are pulled from the
settings
table and env variables are updated by the runtime, to be later consumed by the dbt run execution.
I realize this is not the most elegant or future-proof solution, but I think that dbt transform would be mostly run on the scheduled job/interface which would use cli
style configuration and credentials value passing, and the invocation from the UI(frontend) is secondary use case and can be limited in its capabilities.
@jschatz1 @iroussos
Edited by Melty Bot