Move plugin configuration to the system database
Follow-up of #611 (closed)
We need to implement the backend to the new configuration front-end.
- Update the discovery.yml to add the settings metadata
extractors: - name: tap-gitlab pip_url: "git+https://gitlab.com/meltano/tap-gitlab.git" config: name: gitlab # used as a prefix for ENV version: 0.1 settings: - name: api_url value: "https://gitlab.com/api/v4" label: GitLab API URL - name: private_token label: GitLab API Token description: Either an API token or a private access token. - name: groups label: Enabled groups description: Comma-delimited list of groups ids or name to fetch - name: projects label: Enabled projects description: Comma-delimited list of project ids or path (`namespace/project`) - name: start_date label: Extract from description: Format YYYY-MM-DDTHH:MM:SSZ …
Add a new
PluginSettingstable to the system database
- Store/Update the PluginSettings definitions in the system database on install
meltano.ymlto expose settings per
nameinstead of the definition
plugins: extractors: tap-custom: config: - CUSTOM_SETTING_1 - CUSTOM_SETTING_2 - PASSWORD
Refactor the meltano runtime to hydrate settings from the
We also want to keep the ENV configuration alive, so the process for hydrating shall have the following precedence (bottom has more precedence):
PluginSettingsvalue for the current plugin and setting name
- Environment variable with the setting name
That way we'll keep support for local development using a
.env file without any modification.
- Port the discovery.yml to HOCON to split it, then generate a YAML out of it at bundle time.
- Add a
profileconcept on top of the settings to be able to have multiple variations for a plugin.
- Port over
- Port over