Skip to content

GitLab Next

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
Meltano
Meltano
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 500
    • Issues 500
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
    • Iterations
  • Merge Requests 16
    • Merge Requests 16
  • Requirements
    • Requirements
    • List
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Security & Compliance
    • Security & Compliance
    • Dependency List
    • License Compliance
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Code Review
    • Insights
    • Issue
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • meltano
  • MeltanoMeltano
  • Issues
  • #2393

Closed
Open
Opened Oct 26, 2020 by Tobias Macey@blarghmateyContributor

Dagster as orchestrator

Problem to solve

Add support for more orchestrators to improve the options for adoption of Meltano in a production context.

Target audience

Organizations and data teams that have already invested in an orchestrator other than Airflow.

Further details

I am considering setting up Meltano for use at my day job, and I currently use Dagster as my workflow orchestrator so I was curious where in the code I should look to see about adding support for that integration. Also, if you have any pointers about sharp edges that I should avoid, or any upcoming change in the structure or roadmap of the project that would be helpful.

Proposal

I would like to be able to use Dagster as the orchestrator for executing my Meltano pipelines.

What does success look like, and how can we measure that?

Success looks like having a well defined integration with Dagster such that I can easily get up and running with Meltano as a component in a pipeline when I am already using the Dagster/Dagit service in my environment. The likely path is to add a plugin along the lines of what has been built for Great Expectations or DBT so that I can define a "Solid" that will execute a Meltano pipeline definition and provide detailed output that integrates with the existing logging infrastructure. If it is possible to use the Dagster table space as the metadata storage for Meltano for a tighter and simpler deployment, that would be a stretch goal to work toward.

Assignee
Assign to
Backlog
Milestone
Backlog
Assign milestone
Time tracking
None
Due date
None
Reference: meltano/meltano#2393