Translate all tap and target yaml files on the Hub to meltano discovery.yml format
Click to expand (prior description)
Following on the ideas in Hub metadata for plugins should be a superset o... (#196 - closed) and towards the goal of Reduce friction of adding new plugins to the hub (#192 - closed), we will move to a single yaml format and deprecate one of the two formats.
If there's anything Meltano specific that doesn't make sense in a generic definition:
First of all: why? Seems like we should just have a single definition that services the entire community. Special use cases and plugs should probably be avoided, or they should be written in a generic way that works for another orchestrator solving the same problems.If there are any "special Meltano hooks" that we just can't let go of, can they go into something likeextras:
ormeta:
collection that has vendor-specific implementation details? (Similar to dbt.)
Notes:
- Having just a single file layout streamlines the process of adding new plugins and gives us a path forward for the Meltano installation process.
- In this scenario, the Meltano yaml parser will need to support parsing the new standard file formats but this is basically needed anyway.
Considering Utility plugins as a priority in Q2 and Q3, we likely wouldn't be able to fit everything in the new Singer-specific format. We're pausing any new investments into the Singer-specific format for now and sticking instead with an iteration of the meltano.yml
/discovery.yml
syntax. We probably do need to make some alterations/deprecations/improvements on that format, but that format is most compatible with our current definitions, and it is the smoothest path to being able to add plugins directly from the hub.
Blocks Support `meltano add` directly from Hub (meltano#3283 - closed). Does not block. Endpoints exist.
Deeper analysis/comparison of fields in: Hub metadata for plugins should be a superset o... (#196 - closed) (needs to be reversed in directionality, since we're not leaning to using the discovery as primary).
Action Plan
We may need to promote this issue to epic status. Here's a rough estimate of the steps we'll need to perform:
-
Consider adding version
spec to allplugin.yml
files in the same way thatdiscovery.yml
does, or version constraints. -
Create and publish JSON Schema for plugin.yml
format. -
Refactor discovery.yml
to reference theplugin.yml
JSON schema (more DRY). -
Migrate existing files in _data/meltano/
to use a single-variant-per-file format, for instance https://gitlab.com/meltano/hub/-/blob/main/_data/meltano/extractors/facebook.yml would be two files:.../tap-facebook--singer-io.yml
and.../tap-facebook--meltano.yml
-
Translate existing files in _data/taps
and_data/targets
over to theplugin.yml
format. -
Create follow-on tasks to collect settings
data for the top most-used tap and target variants.