Problems with environment variables taking precedence over declared ones
If a plugin configuration option is set to a specific environment variable, but that config option as a default environment variable it can be configured with, and both are defined, Meltano uses the default one rather than the explicitly declared one.
Example:
meltano init demo
meltano add extractor tap-gitlab
meltano config tap-gitlab set private_token '$MY_REAL_GITLAB_API_TOKEN'
export GITLAB_API_TOKEN=wrong-token
export MY_REAL_GITLAB_API_TOKEN=right-token
meltano config tap-gitlab{
"api_url": "https://gitlab.com",
"private_token": "wrong-token",
"groups": "",
"projects": "",
"ultimate_license": false,
"fetch_merge_request_commits": false,
"fetch_pipelines_extended": false
}To me, this is unexpected behavior. I would expect GITLAB_API_TOKEN to be used if I hadn't specified anything for api_url, but since I explicitly told Meltano to use MY_REAL_GITLAB_API_TOKEN I'd expect that to take precedence.
Proposed Solution (AJ)
I think we should deprecate env: CUSTOM_ENV_NAME and env_aliases: [...] in favor of the more explicit <plugin_name>_<setting_name>. I've created #3120 (closed) to track this.
Edited by AJ Steers