Dagster as orchestrator

Problem to solve

Add support for more orchestrators to improve the options for adoption of Meltano in a production context.

Target audience

Organizations and data teams that have already invested in an orchestrator other than Airflow.

Further details

I am considering setting up Meltano for use at my day job, and I currently use Dagster as my workflow orchestrator so I was curious where in the code I should look to see about adding support for that integration. Also, if you have any pointers about sharp edges that I should avoid, or any upcoming change in the structure or roadmap of the project that would be helpful.

Proposal

I would like to be able to use Dagster as the orchestrator for executing my Meltano pipelines.

What does success look like, and how can we measure that?

Success looks like having a well defined integration with Dagster such that I can easily get up and running with Meltano as a component in a pipeline when I am already using the Dagster/Dagit service in my environment. The likely path is to add a plugin along the lines of what has been built for Great Expectations or DBT so that I can define a "Solid" that will execute a Meltano pipeline definition and provide detailed output that integrates with the existing logging infrastructure. If it is possible to use the Dagster table space as the metadata storage for Meltano for a tighter and simpler deployment, that would be a stretch goal to work toward.