Skip to content

Draft: Generate discovery yml

Pat Nadolny requested to merge generate_discovery_yml into master

Closes: #3010 (closed)

  • Added all split out versions of the plugin definitions
  • Update CI to look for anything in the definitions folder instead of just discovery.yml
  • Added plugin_discovery_generator.py which iterates the definitions directories, builds the discovery.yml, then writes it out to the bundle directory where it currently exists. I included a main so that it can be run in the Makefile as a dependency to the sdist command. I also wrote a simple test for it that writes out a sample definition, generates the discovery, then asserts its correct.
  • I also included a yaml_diff.py which is retro fitted from something in the hub that will pull the current discovery.yml from the endpoint and does a deep diff to whats been generated. This is to prove that the split out definitions and the generation process output the same discovery.yml as if we didnt change anything. I removed this because linting was erroring with it.
  • I didnt remove discovery.yml because I think theres other dependencies in test and elsewhere but when we merge this it will become stale if any definitions change.

I wanted to build this to be tested easily and iterated on because we will be expanding the functionality here to accommodate including singer specs from MeltanoHub and could also eventually run a scraping process for all SDK based plugins to auto generate their discoverable settings.

Warning: any changes made to the discovery.yml while this MR is being reviewed will need to be updated before this is merged. Also I'd probably like to remove the yaml_diff.py as a follow on once its merged since its no longer useful.

Merge Request Checklist

  • Link to an issue
  • Include the proposed fix or feature
  • Include and update tests for the modified code
  • Include a documentation change
  • Add a CHANGELOG.md entry in the Unreleased section for any user-facing changes
  • Mention or assign a maintainer

FYI @tayloramurphy - I'm assigning AJ as reviewer but I'd also like you to review and give feedback.

Edited by Pat Nadolny

Merge request reports