Formalize Meltano web API to simplify integration with other workflows/orchestrators
Problem to solve
Right now there is a fair amount of code that needs to be written and maintained for Meltano to be properly integrated with a workflow orchestrator. In order to make it easier to adopt Meltano it would be useful for it to have a web API that is documented and maintained to enable external systems to use that to drive the execution of Meltano pipelines. As part of that API it would be beneficial to expose metadata of the taps/targets, schema, timing, lineage, etc.
Target audience
(For whom are we doing this? Include a persona) The audience for this feature includes software developers who want a straightforward way of adding data integration to their systems, data engineers who want to include Meltano as part of their data platform without requiring deep customization or long lead times to build an integration, and analytics engineers who want to use Meltano without adopting it as their entire interface.
Proposal
Given that there is already a web application for Meltano it would probably be easiest to add the APIs to that code. Alternatively, it might be useful to build a separate API application using something like FastAPI to get out of the box support for Swagger, and simplify the work of integrating with the Meltano Python APIs.
What does success look like, and how can we measure that?
A successful implementation would provide a set of well designed and documented APIs (REST, GraphQL, gRPC, whatever) that provide access to functionality in Meltano that will drive pipeline execution, allow for querying the state of execution runs (current and past), retrieve pipeline details (sources/destinations, configuration, etc.), and metadata about run results for integrating with data platforms (e.g. data catalogs, lineage systems, orchestrators, etc.)